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ALGORITHMS FOR NONLINEAR 
PIECEWISE POLYNOMIAL APPROXIMATION: 
THEORETICAL ASPECTS 

BORISLAV KARAIVANOV. PENCHe PETRUSHEV, AND ROBERT C. SHARPLEY 

ABSTRACT. In this article algorithms are developed for nonlinear n-terre 
Courant element approximation of functions in Lp (0 < p <_ oo) on bounded 
polygonal domains in ll 2. Redundant collections of Courant elements, which 
are generated by multilevel nested triangulations allowing arbitrarilv sharp 
angles, are investigated. Scalable algorithms are derived for nonlinear approx- 
imation which both capture the rate of the best approximation and provide 
the basis for numerical inlplenlentation. Sinlple thrcsholding criteria enable 
approximation of a target ftlnction f te optinlal|y high asynlptotic rates which 
are deternlined and automatically achieved by the inherell| Sllloothlless of f. 
The algorithms provide direct approxinlation estimates and permit utilization 
of the general .lackson-Bernstein machinery te characterize n-terre Courant 
element approximation in terres of a scale of smoothness spaces (B-spaces) 
which govern the approximation rates. 

1. INTRODUCTION 

Highly detailed Digital Terrain Elevation Data (DTED) and associated ilnagery 
are new becolning widely availal)le for lnOSt of the earth's surface. However, al- 
gorithms for effective approximatiol of data of this type are net yet available. A 
primary lnotivation for this work is the developlnent of effective algorithms for 
nonlinear piecewise polynolnial approximation of DTED maps frein a redundant 
hierarchial system over (possibly) irregular triangulations which are constructive 
in nature. Application of the ideas and theory flOln [4] te the resulting framework 
will permit optilnal entropy tree encoding of the elevation data, elmble progressive 
view-dependent refinements which lnay be focused te user-localized regions, and 
permit the registration of similarlv encoded ilnage textures te the surface (see [10], 
[4] for lnore details). 
Our philosophy is that dependable practical approximation procedures can be 
built only upon a solid theoretical basis. Accol'dingly, we have two primary goals 
in this paper. The first is te better understand nonlinear piecewise polynolnial 
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approximation, in particular, to UlMerstand the natme of the global smoothness 
conditions (spa.ces) which govern the rate of approximation. The second goal is to 
develop or refine existing constructive approximation methods for nonlinear approx- 
ilnation which capture the rate of the best approximation and tan be implemented 
effectivelv in practice. 
This paper addresses nonlinear n-terre approximation by Courant elements gen- 
era.ted by multilevel nestcd triangulations, glore precisely, for a giwn bounded 
polygonal domain E C N 2. let (oE)0 be a sequence of triangnlations such that 
each level OE is a triangulation of E consisting of closcd triangles with disjoint 
interiors and a refillellll,ltt of tho in'crions level OEn-1- VP impose some lltild natu- 
rai conditions on the triangnlations in order to prevent possible deterioration, but 
out results arc valid for fairlv generM triangula.tions with sharp angles.  define 
Z := U,20 OE,- Each such nmltilevel triangulation Z gencrates a ladder of spaces 
0 C 81 C    consisting of piecewise lincar flmctions, where ,, (m  O) is spanned 
Iv ail Courant el'nelts ç0 SUl)ported on oeils 0 at the m-th level . 
Utilizing these primal elemems, we cmsider mmlinear approximation bv n-terre 
piecewise liner fmwtims of the form N = j= aç, where  mav conte from 
diren levels mM lwatims. ()ur first gml is to «haracterize the approximation 
spa«es «onsisting f Ml fimctions wilh a giveu rate of aplroxinmtion. For approxi- 
matière in L v, p < , lhis is donc lu [1 Il, wh«re a collection of smoothness spaces 
(called B-spaees) was introdu«ed and ntilized, ht this 1)aper, we develop this theory 
in the more complicated case of approximation in the uniform norm (p = oe). Out 
progrmn consists f the following steps. First. in order to qumttify the approxima- 
tion process, we dvvelop a collection of smoothness spaces B (T) which depend on 
T and will govern the best approximation. Second. we prove compmfion Jackson 
and Bernstein estimates, mtd. third, we clmracterize the approximation spaces bv 
interpolation spa('e methods. 
(htr second and prinmry goM is, by using the B-spa«es mM the related tech- 
niques, to develop (or refine) Mgorithms for nonlinear n-terre Courront element 
al»proximation so that the new Mgorithms are capMfle of achieving the rate of the 
best approximation. In the present paper, we develop three such algorithms for 
-term Courant element approxintation in L v, which we call "'threshold'" (p < ), 
"'trim and cut'" (0 < p  ), mM "'Imsh the error'" (p = ) algorithnts. 
The first step of ea«h of th«se algorithms is a d««omposition step. We denote 
by O the set of all cells (supports of Courant elements) generated bv 
(ç)eo is obviouslv redmMant mtd, therefore, every fmtction f lins infinitely many 
representations of the form 
(-)  =  (f)ç- 
It is crucial to have a suciently ecient (sparse) initial representation of the fimc- 
tion f çhat is being approximated. In our case. this means that the representation 
(1.1) of f should allow a reMization of the corresponding B-notre [[f[[B(OE). Thus 
the problem of obtaining an ecienç iniçial representation of the fimctions is tightly 
related ço çhe development of the -spaces.  achieve such eciencv bv using 
good projecçors into the spaces , m = 0, 1 ..... 
For compleçeness and comparison, we first consider çhe naturM "threshold'" al- 
gorithm for -çerm Courront elemenç approxinmçion, which is valid only in 
0 < p < .. This algorithm simply takes the largest (in Lp) n-terres from (1.1). 
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Usiug the results ri'oto [11], if is easy fo show that the "threshold'" atgorithm cap- 
turcs the rate of the best n-terre Courant element approximation in Lp (p < ). 
The second algorithm, which we call "trim aud cut", originates from the proof 
of the Jackson estimate in [7] and uses the following idca. First, we partition  
through a coloring into a fmnily of disjoint trees " (with respect to the inclusion 
relation):  := U.= Second. we "trim'" each tree by removing cells 0 G " 
corresponding to iusiguificant small terres aoço from (1.1), located near the t, ips of 
the branches. Third, we divide ("cut") the remaining parts of each tree " into 
sections of small "energy". Finally, we rewrite the significant part of each section as 
a linear combination of a slnall number of Courant elements. The resulting terres 
determine the final approxilnmt. b shall show that "'triln and cut" is capable of 
achieving the rate of the best approximation iii Lp (0 < p  ). 
Pivotal in our development is the "push the error" algorithm, the nmne of which 
was coined bv Nira Dvn. The idea for this algorithm appears in [5] and mav be 
roughly described in LOe as follows. For a fixed e > 0. we "'Imsh the error" with 
«, starting ffoto the coarsest level (-)0 and proceeding to finer levels. Nalnely, we 
denote by A0 the set of ail 0  (-)0 such that laol > e (11011 = 1) and define 
0 := 0e aoço. Theu we rewrite ail remaining terres aoço at the next level and 
add the resultiug terlns to the existing terres aoço, 0  (-)1- We denote the uew 
terlns by doço, 0  Or, and select in A1 all 0  (-)1 SllCh that Idol > e. xx contime 
pushing the error in this wav to the finer levels in the lepreseutation of f. Finallv. 
we define our approxilnant bv A := k0 Aj. Thus terres doço with Idol  ¢ are 
discarded only at a very fine level, and hence the error (in L) is  e. 
Of course, this haire "push the error" algorithm calmot achieve the rate of the 
best. approximation. However,  we shall show iii 3.3 and 5. after solne substan- 
tial improvelnents, the algorithm is capable of achieving the rate of convergence of 
the best n-terre Com'ant element aI)proximation in the uniform norm. 
A focal point of our developlnent is the characterization of the approximation 
spaces generated by the best n-terre Courant element approximation in L and 
the characterization of certain approximatiou spaces sociated with the three al- 
gorithms developed, which show that they capture the rate of convergence of the 
best approximation. 
The outline of the paper is as follows. In 2. We collect all facts needed regarding 
multilevel triangulations, local approximation, quasi-interpolmts, and B-spaces. In 
3. we develop and explore the three algorithms for nonlinear n-terln Courant ele- 
ment approximation: "'t hreshold" algorit hm (in 3.1), "t riln and eut'" algorithm (in 
3.2), and "push the error" algorithln (in 3.3). Section 4 is devoted to establishing 
3ackson and Bernstein inequalities in order to study best n-terre Courant element 
approximation. In 5, we show that the three algorithlns capture the rate of the 
best n-terre Courant elelnent approximation and identifv the associated approxi- 
mation spaces  B-spaces. In 6, we discuss some of the main issues of nonlinear 
Courant element approximation.  postpone until the Appendix the proof of an 
impol'tant coloring lemma used in 3.2 for tree approximation in the "'trim and cut'" 
algorithm. 
For convenience, we use the convention that positive constants are denoted bv 
c, c,.., throughout and thev may vary at everv occurrence. The notation A  B 
lneans that ClA  B  c2A. 
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2. PRELIMINARIES 
In this section we collect ail the facts needed regarding nmltilevel triangulations. 
local approximation, quasi-interpolants, and other results whiçh were developed in 
[11] a.lld earlier papers. The essentials are presened for clarity but without proofs. 
2.1. iangulations. By definition 
çan be represented as h« ration of a finite set  of closed triangles with disjoint 
interiors: E = zeN, .  shall alwavs assmne thal lhere exists an initial 
triangnlation  of E of this fi)rm.  call 
a mullilev«l lriaguh#io¢ «»f E with levels (OE,) if lhe following conditions are ful- 
tilh,d: 
(a) l':v¢'ry lev¢,l OE,, is a lmrtiti«m 
AeT,,,  and OE,, c«msists ¢»f «h»sed lrialigles with disjoint interiors. 
(h) l'h¢' l¢,v«ls (E,,) ¢»f ç are n¢'sed, i.c.. OEn+l is a refinement of 
(«) Ea«l li«mgle   , ]las al I'ast lwo and at most 310 children (sub- 
I i'img[(,s) in OE,z+l, wh¢,re .1I  4 is a constant. 
(¢1) l'he valence X,. of each vertex e of anv triangle   OE (the number 
¢,f the trianglcs ff'oto OE,, that share t' as a verex) is at most .OE, where 
  3 is a COllStallt. 
(e) No-ha¢gig-v«rli««s «omtitio: Ne» vertex of anv triangle   çm that 
I»elongs to lhe interior of E lies in the interior of an edge of another 
triangle from 
Xk' denote bv ç',,z the set of ail vertices of triangles from ,, where if v  ç is 
on the bCmndary of E, we inclnde in ç as many copies of v as is its multiplicity. 
With this mderstanding, we set ç= 
 now introduce three types of nnlltilevel nested triangulations which will play 
an essential role in our developments: 

 Locally regular triangulations. \Ve call a multilevel triangulation 2- = 
[.J,n_>0 OE, of E, a compact 1)olygolml (lomain in N 2, a locally regular triangulation. 
or briefly an LR-triangulation. if T satisfics the following additional conditions: 
(i) Thcrc exist constants 0 < r < p < 1 (r  ), such tllat for each L  ç and 
anv child ' of L that belongs to ç. 
(2.2) rl I  I'1  piPI. 
(ii) There exists a constant 0  3  i su«h that for each '. "  ç (m  0) 
with a. common vertex. 
(2.3)   la,,   . 
 Strong locally regular triangulations.  call a nlultilevel triangulation 
T = ,0 T of E, a compact polygonal domain in OE2, a strong locally regular 
triangulation, or simply an SLR-triangulation. if ç satisfies condition (2.2) and also 
the following condition (which replaces (2.3)): 
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(iii) AJfie transform angle condition: There exists a. constant /3 = /3(T) > 
0 (0 < /3 < -) such that if 0  OE (m, > 0) and A  R 2  R 2 is an ane 
3 -- 
transtrm mapping A0 one-to-one, onto an equilateral reference triangle, then for 
every triangle A ¢ T with a. conlnlon vertex with Au, we have 
(2.4) rein angle (A(&)) 2 fl 
where A(&) is the image of & under A and is therefore also a triangle. 

 Regular triangtalations. B.v &'finition a multil«-«el triangulati(m T of E C 2 
is called a regular triangulation if ç sa.tisfies the fi»llowing condition: 
(iv) There exists a constanl ç = ff(T) > 0 su«h that the minimal angle of each 
  T is greater than or equal to fl- 
The remainder of this subsection makes several observations to better mMerstand 
the natm'e of multilevel triangulations. First, it is clear that the classes of LR- and 
SLR-triangulations are each invm'iant ramer ane transforms. X> next observe 
that ea«h SLR.-tlimgulation is m LR-triangulation. but that the converse statement 
does hot hold. loreover. ea«h regular triangulation is an SI,R-triangulation, but 
again the converse is in general false. Çomtcrexalnples are given in [11]. 
Each type of triangulation dcpends on several parameters which are hot com- 
pletely independent. For instance, the paralneters of LR-triangulations are 3Io, , 
r, p, & and # (the caMinalitv of ). We could set al 1 
 = 7, P= 1-randelimi- 
nate these as parameters, but this would tend to obs«m'e the actual dependence of 
the estilnates upon given triangulations. 
We next brieflv dcscribe a siml)le standard procedm'e for conslructing lnultilevel 
triangulations. XX start ff'oto an initial triangulation  of the given compact 
polygonal domain E C N . We then select a point on each edge of every triangle 
   and join them within  by edges to subdivide  into four children. The 
collection of ail such children hecomes the first generation of triangles, which we 
denote bv . We recursively refine in this way to produce succeeding generations 
, ,.... The resulting collection T := U,0 OE, is a multilevel triangulation of 
E. 
It is important to know how the quantities ][, Inin angle(), and Inax edge() 
of a triangle   T mv change as  lnoves away froln a fixed triangle > within 
the saine level or through the nested refinements. Consider the case when ç is an 
LR-triangulation. Then conditions (i) and (ii) suggest a geometric rate of change 
of ]] (at the saine level). In fact, /ho rate is polynomial [11]. Fur/hermore, 
if '.A"  OE (m  1) have a COnllnOl vertex and are also children of some 
  T-I, then, as shoxw in [11], it is possible for ' to be equilateral (or close to 
such), but for " to have an uncontrollably sharp angle (see Figure 1). 

FIGU1RE 1. A skewed cell 
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If2- is an SLR-tria.ngulatioi, the above configuration is impossible, but the trian- 
gles froln 2- still lnay bave uncontrollably shaip angles. In this case, rein angle() 
changes gradually froln one triangle to the adjacent ones. 
For anv vertex v G çm (m  0), we denote by 0 the cell at level m associated 
with v. i.e., 0.,. is the UlliOll of ail triangles from OE that have v as a common vertex. 
We denote by , the set of ail such cells 0 with v G ç, and set  = Um0 oto. 
2.2. Local piecewise linear approximation and qui-interpolants.  d 
note by Hk the set of ail algebraic polynomials of total degree less than k.  shall 
often refer to the following lelmna (see [11]), which estat)lishes the equivMence of 
differcnt norms of polynomials over different sers. 
Lemnla 2.1. Let P  ll. k  1, and 0 < p,q  . 
(a) For ang tr'iaogle  C 2, 
(b) If A aod A' are tu,o trian91cs such tbat A' C 5 and I1 
(c) tf a' c 5 .., la'l  qll ,,,itn o < «, < 1, tben 
) 1 1 
Io tbe above e.rpressions, tbe coo.stants depeod at mo.st on tbe coespondm9 param- 
tiers and tbe constant c. 
l'he no-hanging-verlices condition (e) of triangulations guarantees the existence 
»f ('ourant elements. Namely, for any vertex v 6 ç (m  0) there exists a 
unique Courant element ç0, supported on 0 G , which is the unique continuous 
piecewise linear fllllCtion Oll E that is supported on 0, and satisfies ç0,. (v) = 1. We 
denote 6 :=  := (0)0¢0- We also denote by S the space of ail continuous 
piecewise linear flmctions over T. Clearly, S E S if and only ifs =  S(v)ço.. 
Throughout the remainder of this section, we assume that T is an LR-triangulation 
of E. X shall often use the tbllowing stability estimates for 
Lennna 2.2. Let 0 < q   attd S =  ao ço, m  O. witb coecients ao 
Then for every  G Tre, we bave 

and hence 

IISIl,  (  
OEO,, ODA 

I 
0 
wzth constants of equivalence dependin9 only on the parameters of 2-. 
estimates the q-norm is replaced bg the sup-norm if q -- oe. 

In tbese 

The proof of this lemlna is fairly silnple and ca.n be found iii [11]. 

Local piecewise linear approximation. The local approximation by continu- 
ous piecewise linear functions will be an ilnportant tool in out flrther development. 
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For f E L(E), q > 0, and any A E "Ym (m _> 0), we denote the error of 
approximation to f from S by 
(2.5) (f) := (f, T), := inf I[/- SIIL(a), 
SSm 
where  is the UlfiOn of all triangles rioto OE, that have a vert.ex in common with 
 Quasi-interpolants. The set y of ail Courant elements is obviouslv redundant. 
To obtain a good (i.e., sparse) representation of a given fimction f, we shall use 
the following well-known quasi-interpolant: 
(2.6) Q,(f) := Q,,(f, 7) :=  (f, 
0 
where (f, 9) := lE f9 and (ç0) are the duals of (ç0) defined bv 
(2.7) « :=   
AEOE.,ACO 
9 the "central vertex'" 
with ,e the linear polynomial that is cqual to  at 
0, and equal to ,A at/he o/ber two vertices of  (recall that N, is the valence 
of v). It is easily seen that 
(çe, e') = 500,, for 0.0' E . 
Obviously, Q is a linear projector, i.e., Q(S) = S for S E Sm. It is crucial 
that e E L and e is locally supported. Conquently, Q is locally bounded 
and provides good local approximation. 
Lemma 2.3. (a) If f  Lv(E ). 1 5 q 5 , ad  E ç, m OE O. then 
IIQ,(/)II.()  cll/ll.(n)- 
(b) IfO < ç   and g = e  " P witb P E H2 and m  O. tben 
IIQ.(g)llL.()  clgll.(n), for 
The constats above depend oly on ] and the parameters of 
For a proof of this lenmm, see [11]. 

From the above lemlna, we see that Q,, : L,(E) -- S,, (1 _< l <- oe) is a locally 
bounded linear projector. There is a well-known scheme for extending Q to a 
nonlinear projector Q : Lo(E)  S, for 0 < ri < 1. This is needed for nonlinear 
approximation in Lp (0 < p  1). To describe this extension, let PA.ç : Lo(A)  H 
(0 <   ) be a projector (linear if ri  1 and nonlinear if 0 < q < 1) such that 
IIf- P,(f)ll.()  cE(f,A) for f E L(A), 
where E2(f, A) is the error of the best L.(A)-approxilnation to f froln H2 (the 

linear polynonlials). We define 
.(f) := 

and set 
(2.8) 

Z 11A. 

T,,o(f) := Qm(p,,o(f)), for f  Lo(E ). 
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Clearly, ,.,  
0< r/< 1). 
The lmXt lelnlna, established in [11], shows that Qm and Tm,., provide good local 
approximalions Dom 8. 
Lemma2.4. (a) ff f  L,(E). 1 5 1 x, ad A  ç, m  O. the 
(b) lf f  Lo(E ), 0 < 1 ,, artd A  T, m  0. then 
lift- T,(fl)ll()  c(fl)o. 
The coustats above doperai ozdy on 1 aztd the parametcrs of T. 
The needed convergence of Q,,,(f) and T(f) to f is provided bv tlle following 
res, dt (sec l,emma 2.15 ff'on, [11]). 
Lemma 2.5. {f f  L,(E). th«t 
Now. we apply a well-known schtullo fi,r ohtaining sparse Courant elelnent re 
resentation of flm('tions. b defiue 
(2.9) qm := Q, - Q,n-t aud tre.-0 := T.o - T-L-o. for m R 0. 
where Q_ := 0 and T-l.o := 0. Clearly, qm(f), tm,o(f)  m. 
For a given flmction f G L0(E ), 1  q  OE. we define the sequence b(f) := 
(bo(f))oeo= flore the expression 
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L.,(E) -, 8m is a projector (linear if 1 >-- 1 and nonlinear if 

(2.10) q.m(f) =:  bo(f)9o, m >_ O. 

Using Lennna 2.5. we have 

(2.11) f =  q,(f)=   bo(f)c2o in L,. 
m>_O m>_O OEO, 

If f C L,(E). 0 < '1 < 1. we define the sequence b,(f) := (bo.o(f))oeçm bv 

(2-1 ')) tm,o(f) =:  bo.-o(f)o, m >_ O. 
0E(9,. 

and again by Lemma 2.5, we have 

(2.13) 

m_>0 m_>0 

Clearly. b(-) is a linear operator while b(-) (0 < r/< 1) is nonlinear. 
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2.3. B-spaces. In this section, we include the necessary tools for the B-spaces 
which we need for nonlinear -t, erln ('ourant element aI)proxination. The B-spaces 
over multilevel nested triangulations of IR 2 are introduced in [11] and used there for 
nonlinear n-term Courant element approximation in L»(IR 2) (0 < p < oc). Iu the 
present paper, we shall use the B-spaces for n-terre Courant element approximation 
in Lp(E) (0 < p <_ o), where E is a compact polygonal domain in IR 2. We shall 
put the emphasis on approximation iu the lllliforlll llOrlll (p = Oo). There are three 
t.vpes of B-spaces (skim.v, slim, and fat B-spaces) that w('re introduced in [11] to 
serve different pnrposes. For ('om'ant (,lement apl»roximation, we need the slim 
B-spaces, which we shall simply call B-spaces. 
Throttghout this paper, we assume that "Y is an L/?-triangulation of a compact 
polygonal domain E in IR . Moreover, the B-spaces B(T), with parameter set 
1/r := a + 1/p a.ccording to two specific choices: (a) p = oc and a _> 1: or (b) 
0 < p < .;x and a > 0, will arise naturallv in out algorithms and error estimates. 
These spaces have several equivalcnt definitions, which we briefly descrihe. 

Definition of B(T) via local approximation. We (lefine B(T) as the set 

) l/r 

where SA (f) is the error of L(f=x)-approxilnation (local) to f ff'oto $,,, for A 
(see (2.5)). It is readily seen tha.t If + g  f5 + gSç with < := i,{r, 
and If + s]B; = IfIB for s e &. Hence l" ]Bi is a Selni-norm if r  1 and a 
semi-quasi-norm if r < 1. 
By ThcoreIns 2.7 and 2.9 bclow, it follows that if f  B(ç), then f  Lp(E). 
Therefore. il is naturat to dcfine a (quasi-)norm in B (ç) by 
More generally, for (I < /< P, we &,fine 
(2.1Q Na,(/, ) := II/ll + (lll/v-/a(f))  
Evidently, N«(f, T) = IIflIB(z). Vhel, clear from the context, we use 
. Definition of norm in B(E) via atomic deconlposition. For f  L(E), 
we define 
( 
(2.17) N(f) := inf (IOl-llcoollV 
Y=0eo coco 
where the iIfiInUm is takeu over all represent.alions f = 0eo c00 in L(E). Note 
t.hat the existeuce of such representations of f follows by (2.11) and (2.13). By 

Theorem 2.7 below, 

(IOl-llcooll)  <  implies  Icoo(')l  < , 
0 

and hence f c= Lp(E) and the series oo Ic00(.)l converges a.e. and in Lp(E). 
Therefore, the way in which the terres of the series are ordered is hot essential. 
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(2.18) 
If p = oc, then 

and the convergence in L-(E) implies a stronger (absolute) convergence in Lp(E) 
(- < p). Bx Lemma 2.1, it follows that 
N(f)  inf ((o]l/p]co])) 1/r- 
f:0eo coco 00 
k / 
f=eo c0 

N,» (f)  inf Icol"- 
f=-oeo cotpo 
 Definition of norms in B(T) via projectors. For f  Lo(E ). we let 
(2.19) f =  bo.,(f)ço 
I,e lhc repreSelllatiol ,,f f ri'oto (2.11) if q  1 and from (2.13) if 0 < ç < 1. 
defiue 
(2.2o) x.(f) := (o-llbo«(f)çol)  
0 
and. more generally (in aceordanee with (2.16)). 
(2.21) NO,o(A) := (lOl/P-/°llbo.o(f)çollo)  
By Lemmas 2.1 aud 2.2, we have 
(2.22) No.o(f)  (IAl/-/°llq(f)llL.)y if ç  1, 
(2.23) NQ.o(f)  (IAIVP/°llt.o(f)llL,,¢a)) r if 0 < q < 1. 

and 

(2.24) NQ,,(f)(oe(lOI1/p[bo,,(f)l)r ) 
In the most interesting case of p = 

(2.25) 

 General B-spaces. 

A more general B-space Bq(T). c > O. 0 < p. q < oc.. is 

defined as the set of all f E Lp(E) su«h that 
f=-oeo coco OeO. 2- 1o1<2 -m+l 
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where the fq-nOrln is replaced by the sup-nonn if q = oe. In this paper, we do hot 
need the B-spaces in such generality. 
 Embedding theorems and equivalence of norms. e recall our assump- 
tions.  have 0 < p 5 oe, and   1 if p = oe and a > 0 if p < oe. In both cases, 
1/w := + 1/p (1/w :=  if p = oe). We record estilnat.es and embeddings froln [11], 
along with the necessary lno(lifications, which me necessarv for the development of 
the main results of this I)at)('r. Tho first oml)ed(ling rosult al)t)ears as Theorem 2.16 
in [111. 
Theorem 2.6. For O < r < p or p = oe, r  1. tben for ar sequence of real 
unbers (co)oe. we bave 
(2.26)  I«lç  « I1«11; , 
OGO P - 
whe c depends onlg o . p. and tbe parmneters of 
Theorem 2.7. lf f  Lu(E ) with O < l < p. and NQ.(f) < oe, then f  Lv(E) 
(f  C(E) if p = oe), ad f bas tbe repsetation f 
series com,er9in.q absolut,'ly a.e. i E and in Lp (respeetivelg. in C(E)). 
(2.27) IIfllp  ll lbo.n(f), oll 
wheve c is idependent of f . 
Pvoof. For 0 < p < oe, the result follows from (2.11). (2.13), and Theoreln 2.9 
below. If p = oe, the theorem follows by (2.11), (2.13). (2.25), and the following 
estimates: 
bo,v(f),ço  Ibo,v(f)l  
Remark 2.8. It is easilv seen that Theorem 2.7 is hot true when p =  and a < 1. 
For this re&son we impose the restriction a  1 when p =  throughout. 
Theorem 2.9. The noms ][. ]]B7(), Na,v(-) (0 < V < P), N(.), a,,d NQ.v(.) 
(0 < ç < p), defined in (2.15). (2.16), (2.17). and (2.21). are equivale, t with 
constants of equivalence depending only on p. 
Proof. One proceeds exactly as in [11] (see the proof of Theorem 2.17 of that 
reference) and proves that 
l/l>  (  (lll/v-/'aa(f),)) / 
f=o¢e coo O o 
provided 0 < W < p. To obtain the norm estimate from these semi-norn equiv- 
alences, we use Theorem 2.6 to give I111  c(). çsing this, (2.28), a,d the 
renmrk after the definition of N(f) iii (2.17), we obtain 
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For the reverse inequa.lity, we use Lemlna 2.2. Theorem 2.6, and (2.28) to obtain 
/- ) /P 
(o "bo,(f)o";)  c(o, ,p) (o "bo.(f)o" 
c < 
cl[fHp+«(  
 cfe(z. 
This and (2.28) iml,ly ,,(f)  c[Iflla(z). 
l'he next embedding tlworem of Sobolev type foliotes immediately from (2.18) 
«»1" (2.24). 
Theoreln 2.10. kbr 0 < no < « and ri := 
«ontinmms cmb«ddin 9 
"" T) 
(2.'2) B7)(7) c B ( , 

 Interpolation. \Ve first recall some basic definitions from the real interpolation 
method. ], refcr the rea,l«r to [2] and [1] as gencral references for interpolation 
thcorv. For a pair of quasi-normed spaces Xo, Xt. emhedded in a Hausdorff space, 
the space X0 + X is defined as the collection of all flmctions f that can be repr 
sented as f0 + f with f0  X0 and f  X. The quasi-norln in X0 + X is defined 
IIfl[x+x, := [If[xo+xl + i,,f ]lf011x + I[fxll.x,. 
f:fo+fl 
The K-functional is defined for each f  Xo + X and t > 0 by 

(2.30) /x-(f,t) := Iç(f,t;Xo.X):= inf IlY011x0 +tllflll.xl- 
f:fo-}- fl 
The ïeal interpolation space (Xo, X), with 0 < A < 1 and 0 < q  x is defined 
as the set of ail f  Xo + X sueh that 
(1 )l 
where the Lq-norm is replaced bv the sup-norm if q 
It is easily seen that if X C X0 (X contimously embedded in Xo), then 
K(y.t)  IlYllx for y  x0 a.d t  1, .d. consequently, 
(2.31) ]f(Xo,X,),  ]IYlIx + [2"Xlç(f.2-")] q 
Theorem 2.11. S'appose 0 < p   and fuether assume that bolh o,   1 in 
the case p = .. and o,  > 0 otherwise. Furthermore. let  := (oj + l/p) -. 
j = O. 1. 
(2.32) 
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with «quival«rd hOtms, provid«d a = (1- A)ao + Aa with 0 < A < 1 and - := 
(et + l/p)-'. 

Proof. We shall prove (2.32) only in the case p > 1. For a proof of (2.32) when 
p _< 1, see [3]. 
We shall use the abbreviated notation B  := B(7-) and B% := B- (7-), j = 
(}, 1. Also, we denote by gq the space of all sequences a = (a0)0e of real numbers 
such that Ila'l«. := (E I1 ) '/" < - 
We set / := 1 and non,,alizc the C(mrm,t elemenls iii Lp, tl,at is, II0]lp -- 1. 
we also renorlnalize the duals qS0 rioto (2.7) accordingly. \Ve denote again by 
b(f) = (bo)oee) the sequence ri'oto (2.10) witl, respect to the ,on,,alized Courant 
elelnents. By (2.24), Theorem 2.7, and Theorem 2.9, if f  B , j = 0.1. then 

(2.33) 

f = Z bo(f)2o and 

recalling that thc clemcnts ç)0 arc normalizcd ill Lp. Thc corresponding statemcnt 
holds for functiols f G B a as wcll. 
b shall next employ the following intcli)olatiol theorcm (sec. e.g., 5.1 of [1] or 
[2]) which follows dircctly froln the dcfinition of thc N-fimctional and the llOrlllS of 
the interpolation spaces. Suppose T is a lincar operator which bomdedly maps X0 
into tb and X into }, where (X0,-¥) and (lb, }) are couples of quasi-normed 
spaces as above. Then for 0 < A < 1 and 0 < q  , T boundcdly lllaps (-¥o, X1),q 
illt.o (}b, }),q- 
Xb introduce lincar Opcl'ators Z and P a.s follows: Z is dcfincd by Z(f)o := bo(f), 
0 G . and P is givcn l)y P(a) := oeoaoço, a = (ao)oeo. By (2.33), IIb(f)ll«.,  
cllfllB for f e , j = 0.1. and hcnce Z  B"  g (boundcdly). Bv the 
above-lnenIiolmd interpolat ion t heol'eln. 

(2.34) 

Z. (B °,B)«  (go,()« (boundedly). 

Similarly, if a G (,-, then bv Theorelns 2.7 and 2.9. we mav conclude that 7)(a) L,,= 
Y0eO a0ç0 is well defined. So if we set f = 7)(a), then 

[[P(a)['B <_ c inf [[(co)o,,e. <_ c,,a[[t.. 3 = 0.1. 
f=}2oee cooo 

Thus 7 ) " g'5 --' B (boundedly). and by interpolati(m 

(2.35) 

7)" (gro" ç,)),r - ( B'°. B*))« (boundedly). 

Finally, we recall the well-known interpolation result (see, e.g.. [2], [1])" 

(2.36) 

(Otro,(rl),Mr = tr, where 1 _-- 1-k _1_  with 0 < A < 1. 
T T 0 T 1 

Clearly, (2.32) follows by (2.33)-(2.36). 

 Skinny B-spaces. The skimy B-spaces were introduced in [11] and used for 
characterization of nolflinear (discontinuous) piecewise polynolnial approxilnation 
on IR 2. \Ve next adapt that defilfition to the case of approximation on a compact 
polygonal dolnain E C ]R 2. Supposc 7- is a multilevel nested triangulation of E 
whicl, additionally satisfies condition, (2.2) (sec §2.1 and [11]). The skimlv B-space 
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Bk(T), vhere k _> 1 and c and "r are as above, is defined as the set of ail f  Lr(E) 
such that 
) l/r 
(.aT) s(, :: (ç(-(s.))" < , 
where w(f, ) is a kth modulus of smoothness of f in L(), defined by 
wk(f,)r := sui, II(f.)ll.() 
and (f,-) is the kth difference of f. The norm in (T) is defined by 
* Fat B-spaces: The iink to Besov spaces. Sul)l)OSe T is an SLR-triangulation 
of a compact l)olygonal donmin E C 2. Similarly as in [11], we define the fat B- 
spacc (T), whcre k  1 an(1 a an(l r are as al)ove, as the set of all fimctions 
J  L(E) such tha.t 
(-asl II(z := (ll-(I.a))" < oe- 
thcorcm, it rca(lilv fi,ll(,ws that Cl2(f, L)  &(f)  c2w2(f, &)r, and hence 
]f8/=(7)  clfIBv(T)  clf]=(7). The st)ace 2(T) is a natural candidate to 
rcplace B (T) in nonlilmar -tcrm Courant element approximation. This is, how- 
evcr. only possible for suciently snlal] a (0 < a < a0). Otherwise 2(T) is too 
"'fat" and cannot do the job. Finally, we note that if T is a regular triangulation 
and 0 < a < k, thcn a(T) coincides with thc Besov space B(L). For a more 
complete discussion of this and other related issues, see [11]. 

3. ALGORITHMS FOR n-TERM COURANT APPROXIMATION 

 Decomposition step for ail approximation algorithlns. The first step 
of each of the three at)proxilnation algorithms that we consider in this section 
is a decolnl)osition step. This stcp is hot trivial, since the set (I'7- := (0)0eo 
of ail Courant elements is redundant and. therefore, each function has infinitely 
many representations using Courant elements. For each algorithm, it is crucial to 
have a sufficientlv efficient initial representation of the fimction f that is being 
approximated. This means that the representation of f should allow a realization 
of the corresponding B-norm. 
To construct the initial representation, we consider two cases of metric approxi- 
mation. If the approximation takes place in L u, 1 < p <_ :x, we utilize the decoln- 
position of f via quasi-interpolation from (2.11) with 1 <_ q < p, while if 0 < p < 1, 
we use (2.13) with 0 < 1 < p. In both cases, we bave an initial desirable sparse 
representation of f of the form 
(3.1) f = Eb°ç°' bo = bo(f), 
which allows a realization of the B-norm (see (2.24) (2.25), and Theorem 2.9). For 
the remainder of this sectiou, iii order to more easily track the dependency of the 



NONLINEAIR PIECEWlSE POLYNOMIAL APPROXIMATION 2599 

constants appearing in the inequalities, we redefine Il fllB(ï) by 
(3.2) II/IIBa() 
which is an equivalent norm in B (T) (see Theoreln 2.9). Without loss of generality, 
we may aSsulne (when needed) that there is a final level (-)L (L < oe) in (3.1). 
3.1. "Threshold" algorithln (p < oe only). In this algorithln we utilize the 
usual thresholding strategy used for o-erm allroxilllation ri'oin a basis in Lp (1 
p < oe). The resulting procedure performs extremely well. due to the sparse re 
resentation realized bv the first step. k" note, llowever, that the derived error 
estimates involve COllstants that depend on p and becolne Ullbounded as p 
The "push the error'" and "trim and cut" algorithlns desçribed later in this section 
will be shown to achieve the correspollding estilnates for the Ulfifornl IIH'III (p = ). 
For this subsection we therefore assulne that f 

 Description of the "threshold" algorithm. 
Step 1. (Decompose) Wc ue the dec(,nll)osition of f E Lp(E) froln (3.1). 
Step 2. (Select the  largest terres) b order the terres (boo)o6o in a sequence 
(bo2 ço) so that 
(3.3) Ilbolço, ll  IIboçoll  ... 
Then we define the approximant T T n 
.4. (y)v y a= eçe. 
An (f), := 
 Error estimation for the "threshold'" algorithln.  denote the corre- 
sponding error of approxinmtiol of this threshold a.lgorithnx bv 
T T 
The argument used in establishing the Jackson error estimate in [11] may be mod- 
ified in obvious wvs to prove the following error estilnate. 
Theorem 3.1. IffG B(T). >0. 1/ç::+l/p (O<p<oe), then 
(3.a) T 
where c depends on . p. and the parameters of T. 
In 5, we shall need the following result: 
Lemma 3.2. If f : o boço is the decomposition off from (3.1). then 
j- 1 
where (boço)= is as in Step 2 and c depends on . p. ad the parameters of T. 
Proof. Applying Theorem 3.4 from [11] to (bo9o)=,+ immediatelv provides the 
desired result. 

Remark 3.3. As we have mentioned, the main drawback of the "threshold'" algo- 
rithln is that it is hot applicable to approxilnation in the unifornl norm, since the 
constant c = c(c, p) in (3.4) tends to infilfity as p - oe and the performance of the 
algorithln deteriorates as p gets large. The obvious reason for this behavior is that 
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f can be built out of many terres (boço) which have small coefficieuts and are sup- 
ported at the smne location. These terres can pile up to an essential coutribution. 
but the algorithm will rail to antieipate their fiture significance. 
3.2. "im and eut (the tree)" algorithm. The idea of this algorithIn h its 
origins in the proof of the Jackson estimate in [7] (see 5. pages 272-276). The 
approxima.tion considered there is hv wavelets or splines over a uniform partition 
in the unifornl llOrlil.  shall refillC this i(lca to devclop ail algorithm for n-terre 
Colalt cleinent apl»I'OXilnatiol in Lv(E), 1 < p  , over L-triangulations.  
begin with a ln'ief desci'iptiol of lhe algorithm and then elaborate on the details of 
«a«h of the inain steps. 
 Description of the "triln and eut" algorithm. 
Step 1. (De«ompose). ki, us« thé, «olmnon decompositiou of f  Lp(E) given 
ill (3.1). 
Step 2. ( ()ylonize th« colis q[" O into man9eable trees ").  develop an al- 
gorilhm (pro«e«Iure) for ««»l»ring Ihe cells »f (-) in such a wav that the 
colis of the smne colon" fi»rm a Ir«e structure as deseribed in Lelmna 3.4 
I»«l«»w. This organizaIim greatly simplifies Ihe lnanagement of the es- 
limaIes, b«»lh Ihe aI»proximalion coustru«tion and the enumeration of 
"active" Courant elelnents in our ai)proximant. 
Step 3. (Trim «ach tr«e). Sin«e all the elements lnay iuitiallv affect the B- 
spaee n«»rm of a flmction, we need to preprocess each tree by pruning 
alI t»ranehes whieh mav havc manv leas, but do hot make a significant 
contribution to thc norm of the flmctiou f, X do this bv running a 
stopping tilne argulnent from the finest level to a coarser level, until a 
significant cmnulative comribution is met.  prune the branch just 
below that element. 
Step 4. (Partition the remainin9 trees into "'segments"). We continue to par- 
tition the remainders of each of the Iç trees bv cutting them at each 
of the joins of branches to form chains rioin the tree. We will easilv 
be al»le to track the numi)er of chains produced bv this procedure. A 
second stoI»ping time argument is then applied to eut the chains iuto 
 "segments'" in ord«r to coutrol the uumber of significant eleluents added 
to the approximant (at most N0 + 1 from each segmeut) and to guaran- 
tee that the culnulative effeçt of the left-over elements (i.e., error) tan 
be controlled bv the final Step 5. 
Step 5. (Rewrite the "'se9ments'" fo control error). Here each segment is rewrit- 
ten at its fiuest level, and its terlninal element (with the new coefficients) 
and some of its neighboring elements are added fo the approximant. 
This allows for a void to be created, so that the residual of the segment 
will have disjoint support with all remaining segments  well as the 
residuals of those previously processed. This insures that. the cumula- 
tive pointwise error remains under control. 

We now describe these rather vague steps in more detail. 
our earlier discussion. 

Step 1 is clear from 

Step 2. In the followiug lemnla, we construct a procedure for coloring the elelnents 
of (3 with K colors u, so that no two Courant elements of the saine color from the 
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saine level have SUl)l)orts that intersect; in fact, cor1espondillg cclls of the sanie 
color will bave a tree structure with set inclusion as the order relation. This allows 
us to partition  into a disjoint milan of sets " (1  u  Iç), and correspondingly 
K 
organize f as the sure f = =lf, where f := 0eo" boço. b tan then 
proceed to process each of the f, without vorrying about its terres from the saine 
level overlapping, and at worst a factor of K will came into the constants for the 
estimates that we dcrive. For its proof, sec tllc Appendix. 
Lemlna 3.4 (Coloring lcnuna). For ay multilevel-triaogulation T of E, lhe set 
 := (T) of all cells geterated bg T con be represc,lcd as a .fitite disjoint io, of its svbsets (O"))1 with Iç = Iç(N0, M0) (No is te maximal valence ad 310 is 
the maximal umber of childrn of a triangle la T) such that each 0'" bas a lree 
structure with respect to set inclusion, i.e., if 0 . O"   wzth (0) °  (0") °  O. 
the either 0  C O" or O" C 0 . 
In order to COml)lcte the remaining Stcps 3-5 we must consider two varialions 
iii the details of the algorithln, (lepell(ling on whether p =  or 0 < 1 » <  Thc 
case of thc mfiform mctric is plcscnt(,d in Subsecti(m 3.2.1, while thc case of Lp 
(0 < p < OE,) is giron in Subscction 3.2.2. 
3.2.1. The ca.se p = ,. Fix e > 0 and let e* .-'- , « whcre we recall that Iç is the 
nmllber of colors rcpl'esenting the tlee structures. 
Step 3. Trimming of 0'" (1  oe  Iç) with e*.  triln each ", starting ff'oin 
the finest level  and procecding to the coarsest level.  remove ff'oto " everv 
cell 0 ° snch that 
(3.5)  Ibol  *. 
0C0 ° 
 denote bv F" the set of ail 0  " that bave been retained after completing 
this procedure, and by Fï the set of all final cells in F , i.e., 0   Fï iff there is no 
0  F  such that 0 Ç 0 . Clearly, for each 0   Fï, 
(a.6)  Ibol   for each 0'  0 . but  Ibol > *. 
0C0  0C0  
 denote fr := 0er boço. Therefore 
(3.) II/ -/11  max II  booll  mx  Ibol  *, 
0°F 0C0 ' 0°F" 0C0"- 
K 
and hence, if we set ff := = fr, then 
(a.8) II/-/11  tç* = /oe- 

Step 4. Partitioning lhe branches of each tree F v into chains aod lhe chains into 
"'segments". For each of the tree structures F" (1 <_ , _< Iç). we denote by F the 
set of all branching cells in F" (cells with 1note than o11e child in F') and by Fo" h 
the set of all chain cells in F" (cells with exactly ana child in F'). It is easv to see 
that 
(3.) #r < 
In fact, o11e proceeds bv induction ff'oin the filleSt o coarser levels, associatillg each 
branch cell ff-oin F by a cell froln F}. For each branch cell, there is alwavs at least 
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one meniber of F} still available flore each descendant edge. Only one is used to 
associa.te with the current i)ranch cell, thereby leaying at least one available for its 
Imxt ancestor bran«h cell in that line. 
On the other hand, #Fh may be nmch lm'ger than #Fï, and so we will need 
to process these elenmnts. A collection of cells 01 D 02 D --- D 01 is called a chain 
if ibr j : 1 ..... ! - 1. Oj+ is a child of Oj and 0  Fn, and the terminal cell 
0t  Fï  P.  partition the tree F  into chains. Namely, we start at the coarsest 
level and construçt (maximal) chains whiçh will ternfinate with either a final cell 
(in l'ï) or a branching cell (in F). XX contime this procedure to the finest level. 
Xb nex[ "section" each «hain into segments using e* as a threshold. Namely, if A 
is a chain and A : (03 )=1 with 01 D 02 D "-" D Or, then we start frOlll the coarsest 
eh'ment 01 mM smn the «oecients of each cell, moving to the next child of the 
«hain mtil the sure exceeds the threshold. At this point we cut the chain to form 
the first (significalt) segment and start this procedure again with the next child in 
line mitil this is hot possible (i.e.. ending without the threshold being crossed). X 
«ail this type of segment a "r«mn«nt segment". Therefine, this procedure cuts A 
into disioint segnwnts a »f thc finm (OjV +u 
 ,j=i, I t  O. so that each segment satisfies 
exactlv one of the following conditions: 
(a) a «onsists of a siugl« «significant et,Il"" 

(3.IO) 
(6) 

13.11) 

(3.12) 

[bo, ] > e* (case ,,f p = 0), 

 IV, I > * (c.e of # > 0), 
j=i 

I 
, Ibo, ' < 

i+p 
\Ve denote bv Z ' the set of all such seglnents o- = (Oj)j= i 
procedure. 

resulting [rOlll this 

Step 5. Reu,riting elcments from certain segments of '. Let a = (Oj)j= 1 be alw 
seginent ffOln Z, and suppose that the finest cell Op of 
rewrite the ('OUlalt elelnents 
()=l boço) of the segment at its finest (m-th) 
hwel. finding coecient.s (co) such that 
 COçO : bojço on 0. 
OOm,O°O,#O 1=1 
XXdenote.« := {0  O : 0 0 # 0and0 C 0}. Obviously. ifa = 1 (i.e., 
the segment consists of a single cell), then the coecient relnains ulmhanged and 
 = a = {01 }. Observe in any case that #  N0 + 1 and Uoex o c o. Finallv. 
set  := =1 , and correspondingly define 
(3.13) AZc(f) := 
aEN OEA 
as our approximant produced by the "trim and eut" algorithln. 



NONLINEAR PIECEWISE POLYNOMIAL APPFIOXIMATION 2603 

 Error estinlation for the "trilll and cut" algorithm (case p = oc). 
Suppose that the "triln and cut" procedure has been applied to a fmlction f with 
4TC( ) 
 > 0, and . e -f = 0A c090 is the resulting al)proximalt from (3.13), wheIe 

A« = U«ez ";» We denote 
,,(e) := ,,±(«):= #A, 
and 

TC 
A.()(f) := IIf - AyC(f)[[, 

ln TC(f)oc := illf { TU } 
A()(f) "n(e) _< n . 

Note that each of these quantities depend ilnplicitly Oll 7. To COlnl)lete Ollr results 
for the "'tliln and cllt'" algorithln, we show fiI'St in Lelllllla. 3.5 that this is a good 
approxiInation to f, and then that the lmnfl)er of elelnents that are used in the 
approxillmnt satisfies the correct estinmtes (see Theol'eln 3.7 below). 

Lennna 3.5. Suppoe tlat ATC(f) is tle appro.rirnant for f given in equalion 
(3.13) whi«h bas been constr«'ted using the "'trm ood c,l'" algorithm. Then 

(3.14) 

Proof. Followil,g the definitiol, (3.13) of AyC(f), we (l('fille 

A":=  coço. 
c,'E" 0 A', 

Then obviouslv. AC(f) = Eu=l/ A". SillCP g* - 2-,e it suces to shoxv that 
In Step 5 we extracted the heart of each segment a = (Oj)j=, added its con- 
tribution to tlle approxilnant (3.13), and cleared rooln for descendant cells. To 
estilnate the sociated error, we introduce tlle ring for a   := 0  0,; then 
R« = 0 when a consists of a significant cell (i.e., condition (3.10) hokls). For any 
llOlmmpty ring « (  "), set ' := (0) and observe that at worst 
L(01) 
It is easv to see that all rings R« (a  ") are disjoint and the set where A" mav 
differ froln fr is contailmd in «e£ 
and then over all colors u, it follows that 
K 
lift rc 
-4 (f)l,    llf- 
This together with estimale (3.8) ilnplies the desired error estimale (a.l). 

Remark 3.6. Conditions (3.5), (3.11), and (3.12) can be relaxed by replacing every 
smn  Ibo[ bv I[  boço Iloe. This would hot change the rate of approxilnation but 
lllay ilnprove the constants in a practical ilnplenmntation. 
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Theorem 3.7. lffEB(T),a_> 1. ç:=l/a, lhenforeachs>O, 
(3.16) TC 
A.(«)(f)oe _ ¢ and n(¢) 
whcre c = c(Sç, ,lv, «). Therefore, 
Pro@  have ah'cadv shown in Lelmna 3.5 that NTC  ¢ < ¢: so we onlv need 
to estal,lish («) 5 c e-'llflla().  first obs,rve that it is cnough to estimate 
VEr, since «ontributions to thc approximmt oc«ur onlv as ea«h segment from 
is processed. Note that at most olp clement is «ontributcd for segments consisting 
»fa singlc signifi«ant eell (3.10) and at most o + 1 «ontributions for the segmets 
satisfying instead cither (3.11) «)r (3.12). 
I ordcr to estilnatc #Eu we first estilnate #F Ï, sin«e it will estilnate certain 
Icrms. Thc stq)ling criterimn (3.6) in Stcp 3. 
t3.1s) * <  Ibol. 
0C0 o 
must hold t«»r each 0'»  I" Ï. No if wc al»lfly the ç-th power to both sides, use the 
cml)(,lding of thc scqu«,nc, spa«es (ç  1), smn over all 0  Fï, and observe that 
Ihe snl)l»orls of th« ('«lls in I') havc di}oint interiors, thon we obtain 
(3.)) #I'ï (e*)  <  
l'hc rightmost incquality follows immediatclv bv our dcfinition of the nornl of 
B;'(ç) (ce (a.2)). 
To conlplete the proof of the theorem, we OlllV nced to cstablish a similar estimate 
tÇr Ihe lmmbcr of elcmcnts of E u. Rc«all, hoV(,l', that the segments « are %rmed 
as disjoint segmcnts of cells ff'oto thc tree structure and eome as one of two types. 
Esig. thosc cx('ceding the thrcshold (sec conditions (3.10) or (3.11)) and. Erem. those 
that do hot (sec condition (3.12)). KlOm thc construction it follows that renmant 
scgmcnts terminate with (qthcr a unique final «cll or a torique bran«hing oeil. and 
so bv (3.9), 
(a.0) #Zrcm  #r + #Fï  'e #Fï, 
which has just l)een shown in (3.19) to satisfv the dcsired bomd. 
Thel'C%rc wc arc reduced to estimating 
timating #F Ï (sec (3.18)-(3.19)) mav be elnploycd once again. Indeed. we just 
replace the condition (3.18} with 
(3.21) e* < lbol, 
and use the fact that the segments are disjoint (considcred as part of the tree 
structure), in order to obtain 
(3.22) g (*) <   lb01 r 
Although hot required hele, the following lemma will be needed in 5 and tan 
nov be established using the techniques of this section. 
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Lemnla 3.8. Let f = ft,+ fl, where f = -oE6)boço, .fJ = -oo bô(;o (3 = O. 1) 
 := Ibl ) <  U = o. ) 
with oi  1 and ri = 1/cj. If the "'trim and ct'" algorithm with e = eo + e 
(zj > O) bas b«cnappli«d to f . r«pwsent«d as obove in place qf Stop 1. then 
(3.23) Te 
.¢+,)(f)  eo + 

and consequentlg 
(3.25) c(f)  «,,-o + c,,-,,ç], ,, = 1.2,. 
u,ith c depending only on oo. t. and th« param«lcrs qf T. 
Pzvof. AIl the clcments for thc proof alrcady appcar in this sui»section, espccially in 
thc proofs of ThCOl'Cln 3.7 and Lclnlna 3.5. and wc hall assume COlni)lctc falniliarity 
with the llot.ation, tCl'lniliology, a.nd estilnates given thcl'C. Denote the mmd)cr 
of cells used in the "triln and eut'" algol'ithlii for (ho), with approximation error 
e, bv ri(e). Silnila.rly, let nj(ej) l»e thc Colrcsponding lmmber of cells uscd for 
f,i (j = 0,1), again rel)reselte, l as ]'J = 0ç), iii place of Step l. çhe 
theoreln will be proved once we establish the estinmte 
(3.26) ,(e0 + el)  2 (,,0(e0) + ,l(e) ) 
for anv «0, «1 > 0. Indeed. bv COlnbilfing this inequality with the results of Theo- 
rem 3.7 (in particular, incqualitics (3.16) (3.17)), ve can see tha.t the estimate 
(3.27) n(e0 + ml)  2ç«rO,Ç TM + 2ç«r1,Ç ri :  
is true if we set e := (ç)l/r?l--1/r, j = 0, 1, where c is the constant appearing 
there. But the fact that n  n(e0 + e) and the dcfilfition of (-) imply 
TC 
c(f),  ¢+,)(f)  e0 + ci. 
Helwe, bv the defilfition of tlle êj, thc rightmost terres of this last ilmquality are 
bounded by the desired terlns on the right-hand side of ilmquality (3.25). 
I1 order to prove estimate (3.26). we only lmed to estilnate the lmmber of seg- 
lnellts  for f. First observe in Step 3 of/he algorithm that for the thresholdilg 
condition (3.6) to hold for f, with z := e0 + e, the «onditiol must also be satisfied 
for that saine cell 0 ¢ for at least one of the fJ with corresponding threshold ci 
(j = 0.1). This shows that the tree F" = F'(f,e) determilmd bv threshold e is 
contained in the union of the corresl)Olding trees F'(ff, eg) (j = 0.1). By the COll- 
struction of segments « from maxilnal chains of F'(f) il Step 4, the Seglnents for 
f are disjoint and one of the conditions (3.10)-(3.12) m,,st hold. If (3.10) or (3.11) 
holds for a segment « of f, then 0« Ib + bl > e0 + e implies the correspond- 
ing condition for at least ont of f0 (and e0) or ri (and Zl). That is, for one of 
j = 0, 1 we must have 0« Ibl > zj, a,d s fl- at let hall of the segments of f 
this condit.ion must persist for a fixed index j (j = 0, 1). The lmmber of relnlmnt 
segments (see (3.12)), on the other hand, may be estimated bv the Stilll of the 
number of remirent segments of f0 and fl, plus the lltlnlber of ?lew bralcllilg cells 
which lna3" arise within the union of the trees of f0 and ll. These new colis are 
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introduced in F"(f, ) when two chains, exclusive to each of the F"(f j, ej), meet, 
therebv dividing the existing chains for each of the trees and creating an additional 
seglnent. It is easy to see that the nulnber of such new branching cells does hOt 
exceed nlill { rï (f°, e0), Uï (fl, gl ) }- 
This accounting of the three quali(ving conditions (3.10)-(3.12) for segments gives 
 lllgX{#(f 0 0) #(fl,l)} + lllill{Fï(f 0,0),rf(fu 1 
 #2(f °, g0) + #(fl, gl), 
which iml)lies lhe desired estimate (3.26) a.nd COlupletes the proof.  
3.2.2. The «ose 0 < p < oe. XX' now l'eturl to coml)leting Steps 3-5 in the ce 
lha.t p < oe. The arguments are quite silnilar to the case p = oe in the previous 
sui)section, and we shall use the notation there and indicate only the differences. 
hdrodu('e a new l)a.raln(,ter O, where 0 < ç < p. and fix g > 0. 

Step 3. TrblTniTg of (-)" (1 _< , _< K) witb e. This step is the saine as in Case 1 
(p = oe) wilh (3.5) l'('l)lace(l i)y 
In contrast to lhe case p = ., the error 1£ - fr I1» i 1o longer controlled soMv by 
g. It will (h'l)end Oll the smoothlmss of the flmction f that is being approxilnated 
(see Theorem a.9 bclow). 

Step 4. Partitioning the branches of each tree F" into chains and the chains into 
"segments". We proceed exactlv as iii thc case p = c, repla.cing conditions (3.10)- 

(3.12) by the following: 
(3.29) 
(3.30) 

Ibo, II01 /» > e (cae of p = 0). 

1/0 /i+p \ 1/0 
<_ e, but y'Albo, llOll/") °) > e (case of p > 0), 
j=i 

(3.31) 

% IIo.,-I'/')  _< . 

Step 5. Rewriting elements from certain segments of ". This step is exactly the 
saine as for the case p = 

Error estimation for the "trim and eut" algorithm (case 0 < p < oc). 
Suppose that the "trim and cut" algorithm bas been applied to a function f with 
0 < c0 < p and e > O, as described above. Let ArC(f)p = eeh ceçe, A C . be 
the approximant produced by the algorithm. We denote 
(e) := #A« TC 
, A.(«)(f)p := IIf- AT C(f)»llv  
and 
TC  TC . 
A,, (f)p := mf{A,(«)(f)p n(g) < 'r}. 
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Theorem 3.9. If f G 177(7). where c > 1/0- 1/p a,,d - -- (c + )-1. then for 
each  > O. 

(3.32) TC  
and hence 

Proof. We first estima.te (e). 
inequality of (3.28)) in Step 3, it follows that 

From the stopl)ing tilne critel'itm (the converse 

(3.34) e < ( , (IbollOI1/p)o) /° ( \ l/r 
  (IbollOI1/P) r) (SillCe T _ O) 
\0C0 o 0C0'» 

for ea.ch 00  F Ï, whi('h ena.blcs us to repeat thc argmncnts from the proof of 
Theorcm 3.7 and obtain the estilnatc :/#Fï <_ c IlfllB7(-)- In going ful'thcr, we 
use (3.30) iii a, silnila.r fashion and the a|ove t infer as in the proof of Theorem 3.7 
that 

(3.35) 
This implies the desired estimate for t(¢). 
ATC 
It rcmains to estinlate t}lç error IIf- ¢«)(f)pllp. xx first estimate lift- fllp. 
To this end, we group the rcmoved cells into collections of comparable 7-11ornls. 
We denote by 
E" := {0  O" F" "0  0' for a.lV O'  ' Fu, O' 0} 
t.he set of all cells a.t which a trimmed branch sta«ts. Note that for each 0 
the inequality (3.28) holds. Therefore, we ca.n partition " into disjoint collections 
-j, j = 1, 2 ..... L", such that =" L =" and 
(a.a) 
0°j 0C0  
for ail j = 1, 2,..., L" except possibly for j = L ", when the leftlnost inequality 
may fail. Hence, since the cells Dom E" has-e disjoint interiors, and recalling that 
IbollO, /  HboçoHv, we obtail, 
L 
j=l 0¢E 0C0 ° -- 0°E 0C0 ° 
L 
kj=l 0*eE" 0C0":" 
L  

<_ c(-2P/OeP)/P=c(L")/Pe, 

j=l 
where we used the embedding inequality (2.26). To estimate L" we once again 
exploit the idea used in estimating :/#Fï (see (3.18) (3.19)). Since 0 < r <_ 0. we 
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We use this and the fact that the collections 
_j m'e disjoint to obtain 

(3.38) 

Comlfining (3.37") and (3.38), we oi,tain 

Il.f,,- ff,, IIp _< «(e-TIl¢;ll(r))l/Pe =  ""ll.ï,,ll,/lr, 

and honte i»v standard sulm&livitity estimates for Lp ((I < p < ,OE) we mav estimate 
the SlIlII 

(:.3.) IIf- frllp 

l/p* 

where 1'* := rein{ 1. t'}. 
T(, complete the proof of the the(n'em, we must estimate Ilfr - A'IIp  This 
«liffers ri'oto our earlier arguments in the case p = . which involved the error 
estimate (3.15) over a ring of a segment. For any such ring R« (a  ") we use 
instead the estimate 

where we used the embedding inequality (2.26). From the above, using that ail 
rings {R«}«e:" bave disjoint interiors, we obtain 

(3.4(}) 

lift" - A"llp -< ( ' lift,'- "11,(,,) p )/P -< 

('Olldining (3.40) and (3.35) yields 

and hence 

Ilfc --'ffc(f)pllp 

1]p* 
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where p* := lnin{ 1,p}. From this and (3.39). we oltain the api)roI)riate estilnate 
whiçh corresponds to (3.14) of the case fin p = 
4 TC f 
(3.41) IIf 
Lemlna 3.10. Let f = fo+ fl, here f = oeo boço. f = oeo ço (j = O, 1) 
1)--1 
u,itb j _ > ! o - p 1 (0 < 0 < P) ad r A := 1/(aj +  . j = O. 1 . Furthermore, s«ppose 
the "trim and eut'" algovitl«m bas been applied to f, usi9 the above represetation 
off it place of Step l, with 0 < 
Tbe we bave 
(3.42) re - 
(«+«(I) 

(3.43) u(so -[- Sl) < C S-T°JÇ ° nt- 1 "1 " 
ad. tberefore, 
Arc{t: ) < C(ll-aOJo-t-ii-cjk/'l) il = 1. o, 
(3.44) 
 J P . ..... 
where c depend o'ul9 ot p, 0, a0. 1, atd the pa'rameters of T. 
Pro@ The proof is very silnilar to the proof of Theorem 3.9. and we shall onlv 
indi«ate the differences, using thc notation and ideas from there Those differences 
are in ebtilnating Fï,  and L  (see (a.aa na (a.as)). Fom th stopping 
«riterium (converse inequality to (3.28)) in Step 3. it follows that. for 0 °  F Ï, 
o + 1 <  (IbollOI1/)  
OCO  
1 ( )1 
 +c (Ibll0ll/)  
OCO  
1/ri 

/ )l/r°/ ) 
< c o (Iblloll/p) ° ÷% (Iblloll/V) 1 
\OCO o \OCO o 

where c o := max{1.21/°-1} and we used the fart that To, T 1 <__ O" 

Therefore, for 

each 0   F Ï, at least one of 
0 < C(- (]b]]o,l/P) "rO) 
\0CO 

or 1 < ¢0  (Ibll°ll/;) 1 
\OCOO 

must hold. Denoting by FÏo and Fl the sets of all 0 °  F Ï for which the first or 
second inequality, respectively, holds, we obtain 
O°FÏ oco  
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and hence 
(.) # _< #0 + #> _<« (0a 0 + ï). 
 obtain silnilar (with the saine right-hmd-side quantity) for #E  and L  by 
using the saine argument. The estinm.te for #E  gives the desired estilnate for 
} 111W use estilnates (3.37) and (3.40) iii the proof of Theorem 3.9. with e = 
go + e;, together with the above estilnates for #E and L , to obtain 
(a.46) rc 
A(«+«,)(f)p 5 c(eo + ml) 
ffoin which the desired estinmtc (3.42) fl)llows. The filml estinmte (3.44) is proved 
t)v sel(,cting éj = (2('/t)l/r, which I)v out rcsult (3.43), gives that n(g0+gl)  n 
«llld Ho 
Ac(f)p  A(eo+e)(f) p  «'ll/P(gO +E1)  «" II--a°,ço+ I--al.çq  
whore we Imve used (3A{i) in the s(,('ond in('qualit3ç 

3.3. "Push the error'" algorithm. The i(lea of this algorithnl to oilr knowledge 
first al)I)eared iii [5]. ()lit goal is to a(laI)t this algorithnl for nonlinear n-terre 
('Olll'allt ('lelllellt al)i)roxilnation iii the lllliforlll llOrlll alld perfect it so that the 
resulting algorithln achieves the rate of convergence of the best approximation. 
Iii §a.a.l, we describe the "lmsh the error'" algorithln iii its simplest and nlost 
naive forln. \Ve follow with three exalni)les which illustrate deficiencies of the 
simple algorithm and the types of traps to which it lllay fall prey. In §3.3.2, we 
give our refined version of that algorithm. Throughout this section. ,ve assunle that 
7- = l.J=0 OE,, is an LR-triangulation of solne compact polygonal dolnain E in IR 2. 
where the approximation takes place (see §2.1), and f _ C(E). 

3.3.1. A naive "'push the error'" al9oritbm (p = oc). We t)egin by outlining the 
basic clClllents of the algorithm. 
Step 1 (Decoropose). In this sut)section we denote by Qj(f) the piecewise linear 
contilmous fimction that interpolates f at the vertices I of all triangles froln . 
Clearly f  C(E) cal1 be represented as follows: 
(3.47) f = Qo(f) +  (Q(f) -Qj-l(f)) =:  co2o. 
3=1 0 
where the series converges unifornlly. Ill practice the series ternfinates at some 
finest level a (J > 1), so that 
J 

Assulning that initially f = 0ea coço, t.here exists a fast and efficient, procedure 
for obtaining (3.47). 
Step 2 ("Threshold'" and "'push the error"). Fix e > 0. We shall begin at 
the coarsest level f0 and proceed consecutively through to higher resolution levels 
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O,O2 ..... (9.I- V'e define A0 as thc set of all cells 0  (90 such that [coi > E 
(ll011 = 1), and sel 
0Ao 0o 
Next we rewri[e all remainilg Ierlns coco (0    A0) at [he next finer level 
and add [he resulting terlns to the correspolding terlns froln (coo)oeo. Thus we 
obtain a represelda[ion of f iii [he forlll 
0 j=2 0 
 next process Ille CouranI elelnents at level 1. ç <lefine At as the set of all 
0  #) such that IboI > e, a.nd set A := OE0eA, bobo. AI1 lelnaining terres bobo, 
0  (-)  At, we rewrite at the finer level  and add thc resul[ing terres o the 
corresponding [erlns (coo)oeo:. The representa[ion of f at [his stage is written as 
.l 
(3.48) f = 1 + A1 +  00 +   CoCo" 
Ve COllIillllP in this way until we reach the filleSt (i.e., highest resolution) level 
j. The only lnodificatiol at this finest level is IllaI we discard all telmS whose 
coecieldS in absolute value do hot exceed our Ihreshold paralneIer e. In Ihis wav 

Since only slnall terres (lb01 _< e) at a single (in this case, finest) level m'e discarded, 
they ca.lmot stack up, and we have 
Sollle lnodifications must be lnade, however, to insure that this silnple and efficient 
algorit hln will achieve sparse l'epreseltatiols iii ail asylnptot ically optilnal sense and 
avoid hidden t raps that will result in using too llla.ny terlllS iii the approxiilmtiolL 
We indicate briefly each of the possible pitfalls to keep in nlind, before developing 
the algorithln in fldl in the lleXt, sllbsectioll. 
Trap 1. The interpolation schelne we used to l'el)l'eSeld f iii (3.47) leads to 
diflïculties, since it does hot always lead to sparse represelt.ations. "Ve give here a 
univariate exmnple which lllaV be easilv extended to two diInensions. 
Let E := [-1, 1], and let f be the hat fimction on [ ',1 2__N_]l for N sufficiently 
lin'ge, i.e., f(z = ç(2Na ") with ç(x) := (1 -Jxl)ll[_l.ll(X), x  IR. We assmne 
that 7 consists of all dyadic subintervals of [-1, 1]. Using the interpolation schelne 
described in Step 1 at the coarsest level, we nmst interpolate the extremes at 
-1,0, 1 in order to decrease the L  error. The resulting error after this stage, 
however, is 1 - --v. Proceeding with the haire "push-the-error" algorithm with any 
 < ½ results iii ail index set A with #A  N. However, the best approximation 
is achieved using the single fine scale element ç(2Nx). Therefore, any reasonable 
algorithln that retains  terres in the approximant should give a rate of COllvergellCe 
O( -'r) for any 3' > 0. 
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Trap 2. For a given ¢ > 0 the algorithm as currently described may produce a 
great lmlnber of undesired terlns due to the superposition of a large number of fine 
level nonintersecting terlnS (coço) with a singlc coarse level terre 
(3.49) f=e([-1.l]+o). 
 set  a.s a set of disjoint cells 0 rioto level 2 with 0 C (-5. 5), where 5 = 2 -'. 
It is clear that we tan choose thesc cells for , so that «M = 2 N. At the central 
vertex x0 of each cell 0 we have f(.ro) > e(1 - 5) + Se = e. The "push-the-error 
algorithm'" will pro(luce an ilmcient apl)l'oxinm(iolL since it will hot select the 
('oarse fil'St terre in (3.49) as one might hope. Instead, no such element will be 
('hosen at the ('oarsest h, vel, and thc errol will be pushed. At each successive 
stage the co«'c'ients of the rewritt(,n des('endan Couralt elements for 00 will ail 
again lie I»encath thc threshold and be furIhor rewritlen until all cells are on level 
'2 N At that stage they will be combined with the relnaining terlns in (3.49). 
The corr(,sp¢mdilg ce,Ils will now bave coccieltS that exceed the threshold and 
nmst I)e s¢'l¢'('tcd. In'Oducing at h,ast 2 N Wrms in the api)roximant. As indicated 
al)ov(', a d('sirablc algoriIhln should haro anticipatcd the trap of lllanv sma.ll, finely 
suI)i)Olted clcmcnts Ihat may corne, ai a laie stage, and would have chosen for this 
flmcIion the aI)l)roximaIion (with thrcshokl e) that consists of a single elenmnt, 
namely e ç[-1,1l- 
ap 3. The final examplc is onc that outmalmuvers a quick remedy to Trap 2, 
i.e., meroly thresholding ail snmll terres at Ihe finest level. For a given e > 0. we 
defilm 
f= e [-.11 + Sj ç[0.z-l + eç[0,Z-MI , 
where mi = j2, 5j = 2-1e ' and BI = 2 N. In this example, elelnents are again 
building near the origin, but now aI)pear at lllany levels with small alnplitudes. 
The "'push-the-crror" algorithln will again take no elements at the coarsest level 
and push the error to the llPXt lcvel. Coll[illtling with the algorithln, we are forced 
to take essentiallv ail tOllllS as the approxilnation to the given fllllCtiOll when. 
optimally, only two terres lmed be takcn. 
It is obvious that we can take each of these telnplate exalnples as building blocks 
and i)uild flmctions to cause these problems for ail «, at ail locations and scales. 

3.3.2. "'Push the error" algoritlm in the uTiiform orm (p = oc). In this section 
we indicate the refinements lmeded in order to gua.rantee that the "'push the error'" 
algorithm will achieve optimal rates of approxinmtion. As with the "'triln and eut'" 
algorithm, we break it down into lnanageable steps. 

 Description of the algorithm. 

Step 1 (Decompose). For f E C(E) initiallv represented by (3.1), we lllcy assume, 
without loss of generality, that there exists a finest level )j (J > 0) such that f is 
written as 

(3.5o) 
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Step 2 ( "'Prune the shr'ubs"). In the current algorithm we are hot able to organize 
the cells of O into trees as we did in the "trim and eut" method, since, once we 
rewrite the error on a finer level, a.djacent trees are immediatelv affected and we lose 
the benefit of their intended organization properties. This step of our algorithm, 
however, is analogous to Step 3 of the «trim and eut" algorithln.  fix e > 0 
and let e* := e/2. Oto" goal is, by discarding small insignificant terres boço in 
the representation of f ri'oto (3.50), to prevent our refined algorithm from heing 
trapped by a situation such as that described in "Trap 2"' (sec the haire "Imsh 
the error'" algorithm of 3.3.1). b shall remove such terlns, but insure that the 
resulting uniform error is at most ê* and dcnote by F the set of ail retained cells. 
Iu addition, we shall construct a set F I C I', consisting of "final cells'" in F. 
First, we lmed to introduce ail organizational concept as a replacemcnt for the 
tree structures of 3.2.  shall say (figuratively) that a cell 0 G  sits on anothcr 
cell 0  G (-), if 0 is at least as fine as 0  and its interior (dcnotod by 0 ) intcrseçts 
the interior of 0% Fnrtherlnorc, tbr 0  G (-), we denote the collection of all cells that 
sit on 0  by 
(3.51) Y0* := {0 e O: 0   0 ° ¢ 0 and level(0) k lew'l(0°)}. 
The procedure of Step 2 will bcgin at the fincst level and proceed to the coarsest, 
level by level, coustructing sets F I aud I'. To initialize the procedure we Imt into 
F f ail significant cells 0 G Oj, i.e., such that bol > e*.  place in F anv cell ri'oto 
Oj that sits on a cell from F 
The inductive step proceeds as follows. Suppose that all cells from ) with lcvels 
j > m (0  m < .1) have already been processed.  now describe how to process 
,.  place into F I all cells 0   (-), that satisfv 
(3.52)  Ibol > *, 
and for which there is no 0 G r I from a higher level (i.e., > m) that sits on 0*. A 
cell 0* rioto 
We mv consider the current version of r f as an intermediate (m-th) version of a 
final set for F. Obviously, a cell 0* from O is discarded and hot placed in F if 
(3.53)  Ibol  *, 
and there is no 0 
The procedure is ternfinated after O0 is processed and Step 2 of the algorithm 
is completed. 

The two sets of cells F and Fy (F/ C F C (r)) produced bv Step 2 have the 
following properties, which follow directly from their construction: 
(i) if 01.02  F I and level(01) ¢- level(0.2), then 0ï ç?0_ = 0: 
(ii) for each 0 °  F I, the inequality (3.52) holds: 
(iii) for each 0 °  F, there exists 0  FI that sits on 0 °. 
Vre set .ff := 0er boço and define 
{ bo, if0F, 
(3.54) ao := 0, if 0  6) \ F: 
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then obviously 
(3.55) fr = Z aç. 
It follows fronl the construction that 

(3.56) [If- frll - ¢*- 
Indeed, to see that this estinmte holds, we let D denote the set of all cells 0  6) 
that were (liscar(l('(1 (luring the iml)h'lnentation of Step 2. i.e., T) = 6) \ ['. Let 
.r  E |)e arl)itrary. If .r  [,-J0ez) 0, then x does hot belong to anv cell tllat was 
(liscar(h,(l, and so .fr(x) = f(,r). On the other hand. if x (5 [,-J0ez)0. then tllere 
exists a oeil 0  (5 T tlat contains .r and has coarsest level. Since 0 ° ,cas discarded. 

the in('(luality (3.53) lllllst h()hl. I! follows that 
If(x)- fr(x)l = I Z bo?o(x)l <_ 
OD 

OEYoo 

wh(,r(, wc have l(,rlnaliz(',l our (,h,m(,nts s(, that ]]ç0]] = 1. This verifies the desired 
in('(lua]ity (3-56). 

Stel) 3 (Pu,s'h the error). \Ve n()w l)ro('ess cells of fr with e*, starting fronl the 
coars('st level (-)0 and continuing to fin('r levels. The outcome of this step will be 
an al)t)roxinlant A := A(f) of the fi)rm 

(3.57) 

J J 
3=0 3----00GA. 

where A./ C (-) and Ai will det)end on f. 
\\e use the notation 

.to := {0 (5 O-0 ° çl0 ° ¢ 0 and level(0) = level(0°)} 

for cells fronl the saine level as 0 ° which are adjacent to it. 
We start fronl the representation of ff in (3.55). We define --\o as the set of all 
0  Oo such that f01 > e* (ll011 -- 1), and ve set -\o := [,-J0eÀo ,U0. \Ve denote 

.40 := Z aoço =: Z doço. 
0Ao 0A0 

For each 0 °  6)j, ç0o call be represented as a linear combination of ço'S with 
0  6)j+l- \Ve use this to rewrite (represent) all renmining terres aoço, 0  6)0 \A0, 
at the next level and add the resulting ternls to the corresponding terres aoço, 
0  6)1- We denote by do¢?o, 0  6)1, the new ternls, and therefore obtain a 
representation of f in tlle forln 

f = A0 -I- 

J 
OG j2 0 

Continuing with the next level, we define / as the set of all 0  6)1 such that 
Idol > :, set Al := [,JoeA eo , and define ,Al :--- 0eh do¢?o. As for the previous 
level, we rewrite the renmining ternls doc2o, 0  (01 \ Ai, at the next level and 
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add the resulting terres to the corresponding t.erms aoço, 0 c:_ t92. We obtain the 
following representation of f: 

f = No + A + E 
dodo 
+ 

J 
j=3 0E(gj 

We contilme in this wav tmtil we reach the highest level of cells 19j. At level 19j, 
we define Àj, A j, and `4.j as al}ove and discard all terres doço, 0 e_ (-)j \ Aj. 
We finally obtain our aI»proxinmnt `4 = `4P(f) in the form (3.57). We denote 
J 
A := A« := l J j=0 Ai and À := À := LJg0 Àg, and so `4 = Y'-0e^ doço. 
Since we throw away only elements doç)o with Idol <_ e* at the finest level 19j, 
we have the estimate 

Hic - `411 
O(-) j \ A j 

and hence, using (3.56), 
(3.58) 

This completes Step 3 and with that the description of the algorithm. 
We want to point out an important distinction between the "push the error'" 
steps in the above algorithm and the "naive" algorithm described in §3.3.1. The 
difference is that each time we put a significant terre doço (Id01 > e*) into ,4 we also 
include the neighboring terres (i.e., from the index collection .Vo). This prevents 
our algorithm from being defeated bv a situation like that described in "Trap 3"" in 
§3.3.1. 
 Error estimation for the "push the error" algorithm. Suppose "push 
the error" is applied to a function f with  > 0. and `4(f) is the approxinant 
obtained: `4ff(f) := -'0e^, doço. As in the "trim and eut" method, we use the 
corresponding notation 
n(e) := #A«, P P 

and 

P(f) := P(f,T) := inf{P()(f) "n(e) <_ n}. 
We reinark that if f 6 B(T), then by the Embedding Theorein 2.7 it follows that 
f is continuous. Estimates (3.59) and (3.60). established in the following theorem, 
imply uniform convergence of the "'push the error'" approximants to f and provide 
the necessary rates of approximation bv the method. 

Theorem 3.11. If f c:_ BT(T ),  >_ 1, r := 1/c. then for each e > O. 
(3.59) P  
A(e)(f) _< e and ri(c) < ce-HflIB7(oE), 
where c = 6N. Furthermore, we bave 
(3.60) AP(f) < c-llfllB(» . = 1,2 .... 

with c = (6N O). 
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Pro@ In order to prove (3.59), we first observe that the direct approxilnation 
estimate ,&/x P 
,(«)(f) <_ e follows from inequality (3.58) in the constructioll of the 
algorithm. Therefore it only renlains to show that #A_ _< ce-TIIfl[).(7). Clearly, 
(3.61) A« _< (N0 + 1)(#À), 
and we need only estimate the cardinality of  := . We split À into two disjoint 
sers,/ï'I and/ï,,.. \Ve define/ï,i as the set of all final cells iii A. that is. the set of ail 
0 Ç/ï, for which there is 11o 0' Ç/ï, of a higher level sitting on 0. We set .; := 

\Ve shall lnake l'eI)eated use of the fi»llowing simple lenmla. 
Lelnnla 3.12. Suppose Ad C (-) satisfies the condition that cells front different 
let,els do ot hat,e interiors that intersect. Then each 0 Ç (-) may sit on al most 
No + 1 oeils from . 
Pro@ The silnp]e hyp(»thcsis of thc lonlnla just states that for a cell 09. to sit on a 
«cil 0, it lnust be on the saine lcvel: but lhere can be at most N0 + 1 such cells. [] 

We first estilnalc the nuln|mr o[ clelnents Ç/ that arise as final cells in Step 2. 
r each 0 ° Ç 1"i, we have, by (a.52), 
(3.6oe) e* <  Ibol  (  Ibol) '/r (  ). 
OE'oo 0E'0o 
Clearly. FI satisfies the hypolhesis of Lenmm 3.12 (see Property (i) of FI, which 
is stated following (3.52)). and hence each 0 Ç (-) mv sit on at most  + 1 cells 
from Fi. Using this together with (3.62). we obtain 
llfll(oE) :=  lbol   ( + 1) -   lbof  (. + 1)-(#Fi)(e*) . 
00 O'»F I 0'0o 

which, since "r _< 1. ilnplies 

(3.63) #I'i < 2(N0 + 1)e-rllf]lv(7). 
We next estilnate #]OEI" the lmlnber of final cells for the index t Ç constructed 
in Step 3. Clearly Donl that construction, a ccll 0 Ç Ç lnaV occur onlv if 0 Ç F, and 
hence  C F. On the other hand, from Step 2, for each 0 Ç F there exists 0' Ç F 
sitting on 0. Therefore, for each 0 Ç I there exists 0' Ç F I sitting on 0. But 
satisfies the hypothesis of Lemlna a.12 (with ,M replaced bv I), and hence a cell 
0 Ç F I lnW sit on at lnoSt  + 1 cells froln I- Dom this and (3.63), we have 
To colnplete the estimate for , we must estilnate .. Suppose 0 °  Ç := 
  I, and let 0'  À be a cell sitting on 0 ° with level(0') > level(0 °) md such 
that level(0 ) is the minilnUln of the levels of all cells in  sitting on 0 °. Such a 
cell exists, by the definition of À, but it is possibly hot Ulfique. XX denote by 
Zo the set of ail 0 Ç F which, while "pushing the error" from 0 ° in Step 3. have 
contributed to the terre do, ço,. Due to the lninilnality of 0'. we see that 
(3.65) do, = do,o,(vo,) =  
OE Zoo 
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where t'0, is the "central vel'tex'" of 0'. 
using (3.65), 

Since 0' E À, then Ido, I > e*, and hence, 

(3.66) * < Ido, I <_ Il  bovo,l <_ (r  1). 
0Gîoo 
It is easilv seen that ea«ll 0 G Zo satisfies the folloving properties: 
(a) 0  0', 
(b) level(0 ) < level(0)  level(O'), 
(c) the "central vertex" of 0 lies on 0 °, and hence 0 sits on 
Property (a) follows by ohserving that the support of an elelnent which is rewritten 
at a finer level always «ontains the supports of the COltritmting fille" elelnents. 
Property (b) holds, since . ç A, and hence 11o terres boço with level(0)  level(0 °) 
lllay contribute to do, ço,. Note that it is possible that thel'e are 0 that salisfy 
properties (a)-(c) above but do not belong to 0.. 
Next. we show that each 0 G [" may belong to at lnOSt  + I sets Z0* with 
0*  À,.. Ind«ed. let O  F and sui»pose 0 °  .,. is such that 0  Zoo. In 
the following, we shall use the notation ri'oin al»ove that inw)lves 0 c, but we will 
consider su«h 0  as arbitrarv in . Let .Mo denote the set of ail 0  Ç  such 
that 0 G Zo,. In particular, 0 ° G Mo- k, fix Mo and shov that it satisfies the 
hypothesis of Lemlna 3.12. Indeed. let 0, 02 G -0 from different levels. But this 
ilnplies 0 G Zoj (j = 1,2), and we lllV as well consider 0 = 0 ° and sav 02 = 0 , 
where level(0 ") ¢ level(0). Evidently. level(0 ") < level(O'), ri'oto property (b) 
applied to 0" and 0. 
Bv sylnlnetry, we nlay assume level(0 ") < level(0°). If (0)   (0°)  ¢ 0, thon 
0 ° sits on 0  and hence, since level(0) > level(0°), 0 Callllot be iii Z0, which is 
a contradiction. Therefore. (0)   (0°)  = 0, which verifies the hypothesis of 
Lelnma 3.12. 
Now that Lelnnla 3.12 can be applied to M0, then 0 ( anv other cell froln 
ln<V sit on at most N0 + 1 cells 0* G «0- Therefore, 0 lnav belong to at most 
X + 1 such sers Zo* with 0*  ,.. Using this and (3.66), we obtain 
I[/[IBe()   lbol   (So + )-1   IboV  (; + 1) -l(Aï)(- 
OGF OO,. OGZoo 
Therefore. it follows (recall that ç < 1) that 
I combine this estilnate with (3.61) and (3.64) to obtain the desired estimate of 
#A« in (3.59). Estilnate (3.60) follows ilmnediatelv ri'oin (3.59). 
The following lemlna will be needed iii 5. 
Lemma 3.13. Let f= f0+fl, where f= oeoboço, f = oeobôço (j = 0.1), 
,d  =  +  (u 0 e ), d 
 := Ib$1" < , (j = 0.1), 
where oo. al  1 ad r0 := l/a0, ri := 1/al. Furthermore, suppose that "'push 
the evror'" is applied using the above repreentatio of f , with  := 0 + 1. where 

OGZoo 0 o 
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£0, £1 > O. Tben we bave 
(3.67) 

Aff(o+)(f) _< eo + 1, 

1(£0 q- £1) -- OEr°-A/' -° -[- C-'/'IH; 1, 

Therefore, 

#à; < (No + 1)(#r;) < (No + 1)(#Fï + #F}) 
(3.72) < 2(N0+ 
To complete the proof, we must next estilnate #Àr. For each 0 ° G /r, we define 
0' G/ and Zoo exactly as in the proof of Theorem 3.11. Similarly as in (3.66), we 

and hence ( rj <_ I) 

:/#F _< 2(5o + 1 
)e A.. 

(3.6s) 
wheve c = 6N. Co.nsequently. 
A, (f) < co-«°JV'o + CI'--alI, "11 = 1.2 .... , 
(3.69) P , « = 
Pro@  follow in the footstct)s of the proof of Theorem 3.11.  shall use the 
notation from there, and only indicate the differences as they arise.  denote 
e* := e + e with Q := e/2, j = 0, 1. Estimatc (3.67) is immediate from the 
description of the algorithm. 
It remains to provide estimate (3.68) for the mlmber of terres used in the ap- 
proximation. As iii (3.61), we have 
(3.70) "(g0 +g) := #A 5 (No + 
where we denotc À := , and ÀI and À,. bave the saine definitions, proceeding 
exactly  in lhc i)roof of Theorcm 3.11. Çonfimling as there, we have to estimale 
#F I. For each o  FI" we have, by (3.52) and lhe fact thal 0 < ri  1 (j = 0.1), 
that 
0EY0o OEY0o 0EY0o 
 (  b[r°) 1/r° +(  [b[T1) l/ri" 
OEYoo OEYoo 
Dom lhis, it follows/ha.t, for ea«h 0 ° E F S, al let one of 
(3.n) 4 <(  Ibgl) '/° or «1 < (  Iblrl) 1Ci 
OYoo OEYo 
must hold.  dcnote by Fï and F} tlle sers of ail 0 ° E F S such that the respective 
condition from (.71) hohls fi»r either j = 0 or j = 1. For j = 0.1. we have similarly, 
 in tire proof of Theorem 3.11. 
 :=  ]b]  2 (N0 + 1) -1   ]bl  2 (N0 + 1)-l(F})(«ff , 
0E 0OEF} 0EYoo 
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have 

or 
(3.74) 

 <( Z Iblrl)llrl 
OEZoo 
nmst hold. \Ve dcnote by j,0. and ,[ tlw sers of ail 0   ,. fir whi«h (3.73) and 
(3.74) hokl. rcspe«tively. As in the proof of Thcorcnl 3.11. each 0  (-) may belong 
to at lnOSt  + 1 sers Z0. 0   ,.. Thcrefore, for j = 0.1, 
.¢  lbDI , (No+ 1) -1   Ib$1 '  (q, + 1)-1(#)(¢) r', 

and hence 

Therefore, 

#X{ _< 2(No + 1)e}-rflÇ'j , j =o. 1. 

f 
_< #j,o. + #X,I. _< 2(No + 1)tero.N'oO ..11_ e-TiHïl) " 
This estimate, together with (3.70) and (3.72), implies (3.68) (sin«e No > 3). Esti- 
maie (3.69) follows by usi,lg e := (2c)",,,-"'A/} (j = 0, 1) i,1 (a.cr) a,lO (a.s) to 
obtain n(eo + 51) __ '/l, and so AP(f)OE _< APn(eO+el)(f)oo <__ gO + gl. [] 

4. BEST ?t-TERM COURANT ELEMENT APPROXIMATION 
In this section, we &SSllme that 7- is a locally regular triangulation of a bounded 
polygonal donlain E with parameters ]V 0,/[o, r, p, 6. and :0 (sec §2.1). We denote 
by (I)7 the collection of all Courant elemcnts ç0 generated by 7-. Notice that 
is not a basis; (I)T is redundant. \Ve consider nonlinear n-terre api)roxilnation in 
Lv(E ) (0 < p <_ oc) from 7, where we identifv L(E) as C(E). Ollr main goal 
is to characterize the approxinlation spaces generated bv this approxilnation, with 
emphasis on the case p = oc. We let E,(7-) denote the nonlinear set. consisting of 
all continuous piecewise linear functions ç of the fornl 

' =- Z a0990" 

06.A4 
where .M C 0(7-), #.M <_ n and .Ad may vary with S. 
the best Lp-approxinlation of f  Lp(E) frolll 
an(f, 7)p := inf IIf- Slip. 
s.(7) 
In or(Dr to characterize the approxillation spaces generated by (a,(f,T)), we 
begin in this section by first proving a compalfion pair of Jackson and Bernstein 
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inequalities, and then follow with the usual teclmiques of interpolation of operators 
(see fol" exalnple [6], [15], [13]). 
111 the following, we aSSlllllO iii general that 0 < p 
p =  and a > 0 if p < : in eithcr case we set 1/r := a + 1/p. 
Tlleorem 4.1 (Jackson estinmte). If f  B(ç). then 
where c dçperMs orly on a. p and [he paromelers of T. 
Proof. Estimate (4.1) follows ffoin anv of our constructive algoritlnns as formulated 
in the corresponding Th('orelnS 3.1, 3.7. 3.9. or 3.11. 
Theorenl 4.2 (Berlsteil estimate). If,S'  E(T), then 
(42) IISIIB(r)  
w5ere c deperds only on «. p, and toc parameters of T. 
l'roof.  shall piove estimatc (4.2) only in the case p = OE. For the proof when 
l' < OE, sec [11]. SllppOSe ,%"  Z(ç) alld ,%' =: E0M COCO, where 
and # 5 n. Let A l»e thc set of ail triangles   ç that are involved in ail 
relis 0  cM. Then N = aeA 5"5, wllerc N5 =:   Ps. P a linear polynoinial. 
Evidently, #A  V o #cM  c. 
k s[lall utilize the natural tl'ce structurc in ç induced bv the inclusion relation: 
Each trimlglc   En bas (COlltaills)  .1[ 0 children iii En+l alld Olle parent in 
E,,-1, ifm k 1. Let F0 bethesetofallçsuch tllat 5D5' %rsomeS'A. 
We denote bv I'b the set of ail braiching triagles iii F 0 (triangles with more than 
one child in Ç0) and by I- the set of ail children of branching triangles in ç (which 
lnav or lllav not belollg to Y0)- Now, we eXtPlld F 0 to Ç := F 0 U Ç.  also extend 
A to  := A U Fb U F. hl addition, we introduce the %llowing subsets of F: the 
set Ff of ail filtal triatglcs in Ç (triangles iii Ç COlltaillillg 11o other triangles in 
I'), the set (i'0)y of the final trimlgles in F0. and the set Fch := F  [k of ail chain 
tciatg[es. Note that each triangle 5  Fch has exactlv Olle child in F.  lllaV 
argue as we did %r ri'ces of cells iii (3.9) that the lmmber of branching triangles 
does llOt exceed ttle llllnl]l,r of filial triallgles, #l'b  #(F0)y, alld SillC (F0)y C A. 
t]lell #Çb  Cil. Using tllis, we llave #F 5 6 #rb  cil, #F I  #A + #F  ch, 
and #  #A + #Fb + #F  cl. Içeep in mind. llowever, that #ch can be nmch 
larger thml o. 
kk lleXt est.ilnate Isla() := Ee Il-la(s);, where T := 1/ (see (2.5) 
for the notation). We dellote, for m k 0, S,, := 0e.ee(0) Coço.  shall 
use that. for   
(4.3) S(S)EE = 5(S - Sm)r  IlS- 
.nd, lso, (S)  IISII()- Recall that  is the UlliOn of the collection of ail 
triangles froln the saine level as  and which share a vertex. e denote 
m0 
Evidently, #  3N0 #À  ('* (the valence of each vertex is  5). XXb çonsider 
two possibilities for each &  T: (a) &  , or (b) & 
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(a) If A G H,,, then IL,, D A' for some A' G À ç T,. Using (2.3). we obtain 
Therefore, by summing ovcr ail m > O, we obtain in this case 

&EH rn_>0 AE H, 

m>O 

= cllSll # < c,llSll. 

 l,xl-la,,(S); <_«llSlloE  IAI/IzXl  
Sulnnfing over nz 2 0 in this case as well. we find that 
 cllSllEE,   I'1/11 
OE, 
(.5) s «llSll   s «llSll #  «llSll, 
,X j=u 
where we bave once swit.ched the ordet of summation and used that I'1  11 if 
' is a child of  (sec (2.2)). 
Çon,binin inefllities (4.) and (.5), we obt.ain ISlS,  «llSll., hi«h i 
equiva}ent to (.2). 

where we used that I1',,,11,,(,) <- I1',,,11(< <_ cll*',ll(,\» <_ cllSll, 
applying Lemma 9.1. From the al)ove, it follows that 

and hence 

l<_a<_n,, A) E F«h rn'ir 

IlS- S.llL(.,» = IlS- SllL(» 
< cll(llSIl ÷ II/l/&)) < l.XlllSIIo, 

Note that if/.X_/ E "T, \ F, then Sl,, = S,,l,, and hence IlS - Srll.«,» = 0. 
Suppose Ai E Fc çl OE,,. For each A E Fco, we shall denote bv , (/ # A) the 
unique largest triangle of/ contained in A. Clearly, we have SI,x,\ ' - S,l/x¢\ ' - 
llA\k, "P"b and Sm]A, = llA, -P,, where P_x, is a linear polynonfial. Therefore. 

s(s); = s(s- s,,); <_  IlS- 
j=l 

(b) Let z_X 
j = 1 ..... .x, with ., _< 3N0. We have. using (4.3), 
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We define the approximation space A'(Lp :: Aq(Lp7 -) generated bv the n- 
terre Courant element approximation to be the set of ail fimetions f Ç Lu(E) such 
that 
(4.6) IIf[IA(L,) := IIfllp + k  (nw«(f , T)P) q < OE' 
with the usual modification when q = oe. 
For a fixed LR-triangulation T, we dcnote bv ff(f,t) := ff(f.t:Lp.B(T)) 
the K-funçtional as defined in (2.30). The .]ackon and Bernstein estimates from 
Theorem 4.1 and Theorcm 4.2 yield (see, e.g., Theorem 3.16 of [15] and its proof) 
the following direct and imrse estimates: 

(a.7) 
(4.8) 

ct,(f, T)p < çlç(f , ,,-) 

+ Q_l.(kcr,.(f.T)p) p'_ . p* := min{p, 1}, 

wh('re c d(,pcnds only on a, p. and thc paramcters of 7-. 
The following charaelerization of the approximation spaces 4'(Lp, T) is imme- 
diate from the inequalities (4.7) and (4.8), using the ot)servation (2.31): 
Theoreln 4.a. If o < 7 < a and 0 < q  , then 
A;(,ç) = (,u(ç)), 
with equivalent norms. 
The next result establishes an important (continuous) embedding, which will be 
needed in 5 in order to identiN the approximation spaces (the ones determined by 
the algorithms, as well a.s best n-terre Courant element approximation)  B-spaces. 
Theorem 4.4. Suppose out standin9 assumptios hold. i.e., a > 1 if p = OE. and 
a >0 ifp< . gwelet 1/r := a+ 1/p, thenA(Lp, T) cBT(T ) and 
(4.9) Ilfll(oE)  clIfIIA(L,OE), 
where c depe, ds only on . p. açd the pameters of T. 
Proof.  shall prove (4.9) only in the ce p = oe, proceeding similarly as in [7]. 
For a proofin the ce 0 < p < , see [3]. Suppose f Ç A(L,T), and let 
S Ç Z(T) be such that 
(.10) IIf- sll 5 oe(f. ). 
Since a(f.T)  O, we hae f = S1 + =1(52- - S2--a) with the series con- 
verging uniforml% and hence (ç < 1) 
(.11) Ilfll()  IlSlll() +  II& - &-ll(). 
 apply the Bernstein estimate from Theorem 4.2 to S - S- Ç E2.+l (T) to 
obtain 
Il&- - &-ll()  «oell&  - -111 5 «2((f, ). + _(f,)) 
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and sinfilarly 
IISll( _< c(ll/ll ÷ Ol(f. 
Substituting the above in (4.11), we find that 

5. APPROXIMATION SPACES FOR ALGORITHMS 
Our goal iii this section is to show that the algorithlns that we developed and 
explored in §3 acllieve (il a certain sense) the rate of convergence of the best t- 
terln Courant elellellt approxinlation.  shall utilize the characterization of the 
approxilnation spaces 
Aq (Lp, 
fronl the previous section (ste TlleorelnS 4.3 and 4.4). k" shall denote bv 
A(Lv, T;Nr), .4 (Lv, T:Nrc), and Aï(Lv, T: N) t h« approxilnation spaccs gcn- 
erated bv thc "threshold", "trim and eut", and "'push the crror" algorithms, re- 
spectivcly. Nalnely, f  Aq(Lv, T; N), whcre N is N , A rc or N , if f  Lv(E ) 
alld 
with the usual lnodification wllen q = OE (it is hot quite a norm). 
Theorem 5.1. Let T be an LR-triazgulation of a botuded polggonal domain E 
2. 
(a) If p = OE,,  > 1, and  := 1/a. tben 
(5.) .4(L,ç; ) =.4(L,ç: 
with equivalent "'norms". 
(b) If O < p < oe,  > O. a.d  := (o + l/p) -, then 
(5.2) .«7(L. ç: vc)= «(L. ç:v) 
with equivalent "orms". were "trim and cut'" is applied with parameter   ç < p. 
A Tct ç and A(L; A) 
Pro@ (a) Let p = .  let A(f) denote 
denote the approximation space generated by the corresponding algorithm. Suppose 
Ilfll(;) < . Eie,ltly. «,(f)  (f), ,,d h,l«. ,i,lg Th,-m 4.4. 
It renlains to show that if ][f]] < oe, then 
(5.3) lfll(;) 
For the proof of this estimate, we shall employ Lemmas 3.8 and 3.13. Since thev 
are identical, it does hot matter if we prove (5.3) for "push the errof or for "trim 
and cut". 
Suppose f = ¢ beçe is the representation of f tllat is used wllile "push tlle 
error" or "trim and cuF' is applied. We ha.ve 
Ilfll := ( IV) 1/.  := /,  > 1. 
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Next, we use a well-known interpolatiol technique. We choose o0. cq. fo, and r as 
follows: 1 = ni < c < o0 and T0 := l/a0, T1 := 1Cl. Hence 0 < r0 < r < ri = 1. 
Now let (Ibo 1)=1 Se the decreasing rearrangenwnt of the sequence (Ibol)oeo, i.e., 
indexed so that 

(5.4) 

lb0,1 > lb021 >--- 

\Ve fix oe >_ 0, and dcm,te f0 := =1 boo and f := Ej=2,,+I boc2o. In going 
further, we apply Lemma 3.8 or Lelmna 3.13 to f = fo+ fl, froln above, to obtain 

2 v 

+«2 -  I/%1- 
j=2"+l 

Using propert.v (5.4) and the facts that r = l/c, I < c < c0, and r0 = l/a0, we 
infer 

where we used the well-known Hardy inequalities, nalnelv, we applied the inequality 
froln Lellnlm 3.10 in [15] to estimatc the first sure and Lelmna 3.4 from [6] to the 
second terre. 
(b) For 0 < p < c, the proof of (5.2) is Sillfilar to the proof of (5.1). The ollly 
diffcrence is that file approlniatc roles of Lclmnas 3.8 or 3.13 are noxv played by 
Lemmas 3.2 or 3.10. We omit the details. [] 

6. CONCLUDING REMARKS 

Our t)rinmry goal iii the I)resent article is to quantify the nonlinear 1i-terlll 
proxilnation ri'oin Courant elements and use it to develop algorithms capable of 
achieving the rate of the best approxilnatiol. This is closelv related to the funda- 
mental question in nOlflinear approxilnation of how to lneasure the smoothlmsS of 
the fimctions. As we show in this article, for n-tcrln Courant elelnent approxilna- 
tion when the triangulation 7- is fixed, it is natural to lneasure the SllloothlleSS via 
the scale of the B-spaces B (7-). The use of these spaces allows one to characterize 
the approximatiol spaces for ally rate of convergence  > 0. It also enables us to 
develop algorithms which attain the rate of the best approximation. 
It is natural to add another degree of nolflinearity to the approxilnation bv 
allowing the triangulation 7- to var5". Thus a flmction f should be considered 
smooth of order c > 0 if infT-II/llB() < ec, where the infinmll is taken over all 
LR-triangulations 7- (with fixed paralneters). Therefore the rate of n-terre Courant 
elelnent approximation to f is roughly 0(-). Sumlnarizing, our approximation 
schelne proceeds as follows: (i) for a given filllction f. find a triangulation 7- and 
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a B-space B'(Tf) in which f exhibits the lnOSt Sluoothness, (il) find an optimal 
represeutation of f in t.erms of Courant elements from ç, and (iii) run an algorithm 
that achieves the rate of the best tt-terul CHrant elelnent approximation. The 
first step in this schelne is the most COlnplicated one.  do not have an efficient 
solution for this as yet. Iu the silnpler case of nonliuear approxilnation by piecewise 
polynonlials over dvadic partitions, this problenl, hmvever, has a conlplete and 
efficient solution [14]. As we show, once the triangulation ç is determined, the 
relnaining txvo steps are now well understood and havo efficient solutions in both 
theoretical and practical senses. 
The three algorithlns that ve develop and explore in this article provide solu- 
tions of the probleln under appropriate conditious. A colnmou feature of these 
algorithms is the first step, a nontrivial deconlposition flOlU the redundmlt collec- 
tion of ail Courant elellents from 7. After this initial step, however, thev take 
three different routes. The "threshold" algoritlnn is COnlpletely unstru«tured ]rot 
e3" to implelnent. The drawback of this proçedure is tlmt it is uot valid in the 
case of the Ulfitbrnl nornl, and as a consequeuce it does hot perform well in L v for 
p large. The "t.riln and cut" algorithln is wdid for L v, 0 < p  OE, but it is over- 
structurcd and as a rcsult the tel'tbrnmnce suflk, rs. The "Imsh the error" algorithnl 
appears to be the preferred Ul»l»roxilna.tion nlethod. 
The algorithnls that we dew,lop in this m'ticle are hot restricted to t-terln 
Çourant elellellt approxinlation. Thev can be applied ilmnediatelv to the approxi- 
mat ion from (discont iuuous) piecewise al»proximat iou over lmflt i level t riaugula.t ions 
(for the precise setting, sec [11]). Iu this case the role of the B-spaces B7 (T) should 
be playcd by thc skinny B-spuces (T), introduced in (2.37). The results are sim- 
ilar, but sinlplify considerably. We olnit the details. 
Furtherlnore, these algorithnls Call easily be adapted t.o nonlinear -terln approx- 
imation by Slnooth piecewise polynoluial basis flulctions such  those considered 
in [3] and, in particular, by box splilleS. The main diflçrence would be that one 
should use the corresponding B-spaces, developed in [3], but proceeding in a siulilar 
malmer to this paper. 
It is natural to use (wavelet or prewavelet) bases in nonlinear approxinlatiou, and 
specifically for approxinlation in Lp (1 < p < ).  are hot aware of COlnpactly 
supported wavelets {prewavelets) generated bv Çourant elements or smoother piece- 
wise polynolnials on gelmral nlultilevel triangulations. It is clear to us that such 
wavelet bses vould he verv "exponsive'" to construct and hence are of linlited prac- 
tical value. However, iii the case of uniform triangulations, COlnpactly support.ed 
prewavelets and wavelet flalnes generated by Çourant elellents, or box splilleS, do 
exist, and have been implemented in practice. Obviously, the n-terln approxilllation 
fronl such bases or fralnes cannot surpa the rate of the best t-terlll Çourant (or 
box spline) approximation, but they may give better coustants and hence better 
performance results in pract.ical situations. 
It is also an important, observation that. even in the case of uniform triangula- 
tions, the B-spaces used here are different from the Besov spaces used in nonlinear 
approximation. For a nlore complete discussion of this issue, see [11] and [3]. 
Finally, we remark that in a related paper [12] we extend the argunlents of 
this paper to develop a corresponding approach in the Hausdorff nletric which is 
natural for approximating surfaces. There we also consider various practical aspects 
for decompositions, nmnerical approximation, and data structures. 
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APPENDIX. COLORING LEMMA 
In order to kecp focus on the main analvtical results of the paper, we bave 
postponed the proof of thc coloriug lcnmia used iii Section 3.2 to this appendix. This 
decolnpositiolI result was used to create a manageablc collection of tree structures 
fol" estimatiug both the error and the mmibcr of elenients used in our coustructed 
approxilimlit. Silice this is a gelieral purpose result which lnay prove useful in 
similar settiligs, we give its proof iii full in this appcndix. Fol" clarity we ha,ce 
broken down the proof into a scries of lelnnIas. Since the coloring is donc in several 
refinenient stages, itis helpful to think of thc colorilig as an ordered triple of 
primary, secondary, and shadc colors. Elie I)rinIary coloring b'ill sort the elements 
pel'iodically by l'csolution lcvel, tlic secondary colorilig will insure there is spatial 
color separatiolI, and the third colorilIg (shadilig) is a niore delicatc adjustmcnt to 
ilisure that trce structures are formcd. Wc i)egin by rcpeating thc statcment of the 
coh)ring lcninia for the reader's convcnience. 

Coloring Lemma [see Lemma 3.2]. For any LR-triangulation 7- of E, the 
set 6) := 0(7-) of all cells g«nerated bg T can be r«presented as a finite disjoint 
(0)u=l with A" = h (,oE0, «Io) (,oEç is the mimal valence and 
union of its subscts "  "  
M. is the maximal tuntber of childre of a triangle in T) such that each " bas 
a kree structure with .respect to the inclusion lation, i.e.. if 0'. 0"  0", then 
( O'F  ( O"F # O. or O' c 0", or 0" c 0'. 

To begin the proof, we show, without loss of generality, that for the purposes of 
cololing we nIav assmne that the niultiresolution triangulation provides sufficient 
lesolution with each rcfinenient step. \Ve argue below that after a certain fixed 
liUlifl)er of ilicreliieltS of the levcl there will be a guaranteed refilienient of each edge 
and trimigle, which bv hypothesis is controlled fionl above, i.e., tllliforllllV bounded 
valences and nlax litllnber of subtrianglcs for each refinenlent. Consequently. we 
may separate the levels of O into L (L := [12NO 1112 /0]) disjoint classes (prilnary 
colors) by placing two levcls iii tlie Saille class iff their indices are the saine (nlod L). 
Thus a class  is of the fornl ( 11 Uj=0 j, where (0 := Ojo for sonie 0 < j0 < L 
and (j := (Ojo+j L. Since each such class  has a different priniary color, it will 
suffice to show how to dcsigna.te thc secondary colors of the nielnbers of a single . 
Thercfore, to sinipli" the notation aud wording of argunicnts, we will silnply refer 
to (secondary) coloring tlie classes  ilistead of O. In Lenuna A.1 below we show 
however that tliese classes have additional useful pl'operties. Loosely speaking, 
part (a) shows that the old vertices on a given level are far apart iii terlns of the 
graph nietl'ic. Iii part (b) a similar statement is given for the "'central parts" of 
non-overlapping edges of Curant elelnents fi'oni different levels of (. 
For D C ]2 and m _> 0, we define the star Stm(D) inductively by çt°(D) := D 
and Stm(D) := U{0 G o  level o = rn, 0 ° f3 St-I(D) ¢ 0}. For the vertices in 
resolution level m. this is just the neighborhood of radius k in the graph metric. For 
an edge e with vertices v  and v" and au integer m > level e, we define the "central 
2 R--1 ,t 
part" oftheedgetobest(e,m):=Stm(e\Str ({t ,v"})),whereR:=,., o +4. 
This selection for R has been lnade sufficielitly large so tha.t part (b) of the following 
leunna holds. 
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Lelnlna A.1. The Courant collection ) d«scribed above sati.iïes the follou,ing con- 
ditions: 
(a) For ea«h edge [v, v'] the distance between v and v'. rneasured in the graph 
metric on the next finer lcvel of (--). is af least 4R. 
(b) If e and e  are edges frein cells in ). m is an i'nteger with m - L >_ 
level e _> lcvcl e', and e  e'. then st(e, m) ç e' = (3. 
Pro@ (a) Note that each cdge in (-) gcts subdivided at lcast once after 2N0 lcvels. 
Further, observe that al'ter / := 2\ refincmeltS of any triangle, uone of its 
vertices can t)c COlme('tcd te thcir oI)posite edge bv a single edge at the finer level. 
Using this observation repcatcdly, one ca.n verify that after L refinements, the graph 
nletric distance I)etween t, and v  will I)e at least 2 L/l° = ]lI3 Nô > 4R. 
(b) Let v and v  I)c the vertices of e. Using twice the observation frein the 

proof of part (a), we conchlde that the distance frein each of the vertices in 
e \ ,çtl_2No({V,v'}) to e' is at least 4 when lneasll,ed iii the graI)h nletriç on 

level m. Therefore. on the m-th level, e \ stl_l%({v,v'}) has a. buffe, of at least 
three layers of triaugles lhat separates it fl'Olll Ct. O11 the other haud. the existence 
of ]il0 and th« choie' of R gUa.lalit'e that 5t, ({v. v'}) D St_2N({v.v'}), and 
this estal)lishes the «la.iln.  

This COlnpletes the i)rimary colol'ing, and fi'om this point on we onlv need work 
with a particula.r (2) (i.e., a fixed primary color). In this case "level 0"" will now refer 
to the level of 0 in (2) rather than in (-). as will the star St(O) and st(e, m). Also. 
when referring to the color of a. ('ell we will now lnean the secondarv color, mfiess 
otherwise specified. For 0  ) we denote by 00 the I)oundarv of 0. and bv x0 the 
central point of 0. We say that the cells in -(=)' C (2),, are R-disjoint (R >_ 1) if 
0 ° ç St(O') = 0 for anv 0, O'  '. 
The next result is used for the (secondary) coloring of cells of ). proceeding from 
coarse to fille levels, and uses M colors, so that sanie color cells are R-disjoint. 
Lemma A.2. Suppose some of the cells on a 9iven level are colored in ]iI := 
jyô+l _]_ 1 (R >_ 1) colors se that the saine celer cells are R-disjoint. Then the test 
of the cells on that level can ha colored in the saine ]il colors se that the saine celer 
cells are R-disjoint. 
Pro@ Te complete the coloring Oll the given level, we first use celer #1 te paint 
as many cells as possible se that the saine celer cclls are R-disjoint. Next. we 
use celer #2 as lnuch as possible, followcd by the third and se on until either all 
cells get painted or we run out of colors. The latter case, however, ncver occurs. 
Indeed. assume the contrary and let. 0 be the first cell that cannot be colored by 
this algorithm with the ]il colors. The cell 0 has the property that xvitllin its R + 1 
star St+l(o) there lnust be at least one cell paiuted with each of the ,'il colors. 
But this contradicts the fact that ]il was selected te be at least as large as the 
number of cells within ç/Rm+l(0 ). [] 
For the secondary coloring we proceed inductively, begilming at the coarsest level 
0, and color cells in M colors so that sanie color cells are R-disjoint. Suppose 
then that all levels up t.o "-. (/," > 0) have been colored. We color . as follovs. 
Step a) (Color corner cells). First we define the notion of corner cell. A cell 0 
of level k is called a corner cell for a coarser cell 0  if O' has ail adjacent cell 0" (a.t 
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FIGURE 2. ('orner oeils fl'Olll Step a) 

the smne level of course) so that a'0 lies on edge [Xo,. xo,,] and xo is adjacent to xo,, 
on the level /," (see Figure 2). Given a cell 0' E (2)k-1, we color each of its corner 
cells 0  )a. the saine color as 0'. This insures that a cell's color is propagated 
through all fincr levels to its corner cells. 
Step b) (Ex'tend the colorir.q fo R-stars of the vertices on level (k- 1)). For 
each vortex ' on level (k- 1), wc paint tho cells contained in çtR+2rv 
colors so that the coloring donc in Step a) is preserved and ea«h color is used at 
nlost once. This is alwavs possiile, since Al was selected sufficiently large. Note 
that after this ste 1) lhc saine «olor cells are R-disjoint, since part (a) of Lemnm A.1 
guarantees that the stars are sufficiently separated. 
Step c) ( Complete the seco¢tdar'9 colorm9 of 0.). Accounting for the cells 
previously Iminted in Steps a) and b), we color the remaining cells from . as 
described in Lenmm A.2. 
This procedure specifies the secondary coloring of (2). and we have thus repr 
sented it as a finite disjoint llnion Uv=l ), where are all cells (secondarily) 
colored in the -th color. Thus the prilnary color skips levels until sucient refine- 
lnent is guaranteed, while the secondarv color insures sucient spatial separation 
on each level to control cell overlaps. Unfortunately. the collection of saine primary- 
secondary cololed cells (") lnight not fornl a tree structure, i.e., there lnight be 
two cells in "whose inteliors lneet but neither of theln contains the other. This 
lnaV only happen when a fincr cell lies on tlle edge of a given cell. To fix this defect 
we will set for each fixed )" the third coloring component, the hade of the cells, 
from two possible choices. First. we say that 0' and 
edge of the finer of the oeils is contained iu an edge of the coarser. 
X now restrict our cells to be of fixed prilnary and secondary colors (i.e.. fix 
and inductively determine the shade of these cells. On the coarsest level  of " all 
cells are disjoint, and we assign them shade #1. For the induction step. we suppose 
cells of all levels of "up to level k have been shaded and each shaded collection 
satisfies the desired tree properties. We say that a cell 0 is shade-consistent with 
if xo does hot lie on an edge of any cell that has the saine shade as 0. Hence it is 
possible to place 0 in this shade collection and preserve the tree structure. In this 
case we will also use the terlninology that 0 is consistent with that particular shade. 
We now proceed to shade the cells belonging to level k, i.e., 0  , accordiug to: 
Case i. If 0 both touches and is shade-consistent with some coarsel" cell . then 
we assign to 0 the saine shade as that of the finest such 0. Recall that 
this finest cell is unique by the construction of . 
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Case ii. Othcrwise, we assign to 0 the fil'st lllnllbered shade for vdlich 0 is COll- 
sistent. If no such shade exists, we introduce a new shade for (. 
Bv the constructiol iii the induction step. it is obviols that cach shade sub- 
collection has the desired tree structure. \\'c will show that these criteria intro- 
duce at lnost two shades. For this we need a couple of technical facts. \\'e relnind 
the reader that all cells bclong to a fixed )', i.e., thev havc a fixed prilnary and 
secondarv color. 

Lemma A.3. If (4 iutersects au ed.qe c' of a coarser cçll O' btt is ,ot oue qf ils 
Pwof. Let e' be an edge of 0' that intersects 0. and let , be a vertex of c'. Bv 
Step b) of out coloring procedure (for secondaly colors), St(t,) contains a ceci'ner 
cell 0" iii ),n that is shaded the Saille as ' . BV Step c) in the COllStlllCtiOll of 
St(v) does hot contain any other cells ff'oin ). Since O is hot a cOrllt,r cdl of 
R-1 
then 0 # 0". Therefore #  SI,, (t,) = , and so O nmst lll('et gl 
where t,' is the relnaining vertcx of e'. Th'rcforc, O C st(e', ). 
Lemlna A.4. Çell of )" ,ith di>rettt atades do ,ot totch. 
Proqf. Suppose to the contrarv that cdls 0 a, 0.  " of different shades (shadc 
shade #k, respectively) do touch. > mav rira assume that Oj is a nmxilnal (i.e.. 
coarsest levcl) cell of shade #ff that touches 0t., and conversely, that 0t. is a nlaxilnal 
cell in shadc #k that touches Oj. This follows bv iteration and the fa.ci lhat there 
are onlv finitdy manv COal'ser levels; so the iteration lnUSt terlninate. 
 lnay assume without loss of generality that lcvel Oj < lexx,1 Or. =: ttt-, and 
let ej.ck denote the edges of 0j.0t. respectixwly, such that ek C ej.  consider 
the two cases under which the finer cell 0t. could bave been shaded, and show that 
each one leads to a contradiction. 
For Case i. In this event there would be a coarser cell   " of shade #k that 
touches 0k and to which 0t. would be sha.de-consistent. Le êk be an edge of 
where it is touched by 0k. > consider two possible subcases, depending upon the 
relative level of  to that of Oj. 
Subcase i.a k is filler than Oj. 
Since levd Oj 5 level t. < level 0, then by part (b) of Lelnlna A.1 either èk C e 
or st(êk, mk)  ej = . The first possibility nmy be ruled out, since it would ilnply 
that the coarser cell t. would touch 0j, but 0k is the lnaxilnal such cdl of shade 
Hence st(êk, mk) lnllSl be disjoint fronl ey. Note that Or. is 110[ a conler cell of . 
If that were the case, then 0k would be disjoint ffoin the interiors of all edges on 
level  except the edge on which «0 lies and the edges (at lllOS[ tWO, possibly one) 
where t- is touched by 0t.. Hence, ej lnUSt overlie one of the these edges, since it 
contains e. This, however, contradicts the fact that 0j touches 0k in the forlner 
case and contradicts the inaxilnality of 0k in the latter. Therefore 0k CallllOl be a 
corner cell of , and so, by Lemlna A.3. 0 c st(è«..,). But we have already 
proved that st(êk, mk)  ej = , which is ilnpossible, silce 0k touches 0 
Subcase i.b k is coarser than Oj. 
Since level  < level 0j < level 0k, then again by part (b) of Lelnma A.1 either 
ej C èt. or st(e, ttz)êk = - The former case contradicts maxilnalitv of 0t. relatix 
to 0 a. For the latter cse, note that 0k cannot be a corner cell of Oj, because 
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and Oj have difforent shades. Therefore, by Lemma A.3.0,. C st(ej, mk), and so we 
obtain 0k  êk = , which is impossible, since 0k touches  on êk. 
For Case il. If lris case occurred for the shading of 0k, then since 0j is both 
coa.rser than and touches 0, 0 must hot have been shade #j consistent. Hence 
there lnUSt be a Oj  " of shade #j that is coarser lhan 0k, and «0 belongs to 
some edge ê of j. X consider two possible subcases, depending upon the level 
of j relative to that of 0. 
Subcase ii.a Oj is coarser than Oj. 
Since level 0a  level j < level 0», thon compare edges e, êj using part (b) 
of Lenmm A.1 to infer eilher st(ê,mt-)  ej =  or êj C ej. In lhe latter ce, 
it follows that both the edge ek (rocall Ok touches the coarser Oj on ek) and the 
opposite vertex 0 (since »  ê) of a triangle in  are contained in e, which 
is clear b" impossible. If the former case holds, i.e., st(ê, mk)  e = , then a 
contradiction also results. To see this, observe that 0 cannot be a corner cell for 
a, due to the fact tiret they have different shades. But Lelnma A.3 implies that 
O C at(êj, mb-), which contradicts the fa«t that 0  e # 0. 
Subcase ii.b Oj is finer than Oj. 
Since level j < level Oj < level 0t-, we again compare edges èj. ej /lsing part (b) 
of Lemnm A.1 to imply either .st(ej, mg)  èj =  or ej C êj. By quite similar 
algmnent.s t.o the previous sui)case we can in'ove that contradictions are reached. 
Specifically, the latter statenmnt implies that both the central vertex 0» and its 
opi)osite edge et. belong to the edge êj. On the other hand. the fact that 0k cannot 
be a corner cell %r Oj will imply that Or, C st(ej, m), which will show that 0 
belongs to the intersection st(ej, mk)  êj, and contradict the former statement 
al)ove. 
By out smnption that different shaded cells could touch, we are led in ail ces 
to contradictions, thereby completing our contrapositive proof. 
By combilfing the previous results with the next lenmm, it follows immediatel3 
that  can be colored with K := 2ML colors, and the proof of the coloring lemma 
will be complete. 
Lemma A.5. Af most two shades are reqaired. 
Proof. Suppose in Case ii of the shading step a.bove that a third shade were needed 
for some cell 0. Then its central point 0 ¢ e  e2 for sonle edges e of 0 and 
e2 of 02, where 0, 02 ¢ " are coarser than 0 and bave shade #1 and shade 
respectively. Now, if «0 were a vertex for e, then there would be a corner cell of 
0 in " adjacent to 0. which is clearlv impossible, since cells at the saine level are 
H-disjoint. The saine reoning a.pplies to e2. Therefore 0 callnOt be a vertex for 
either e or e2, and we conclude that eï  e ¢ . Hence, 0 and 02 touch, which 
contradicts Lemnm A.4. 
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THE ALMOST-DISJOINTNESS NUMBER MAY HAVE 
COUNTABLE COFINALITY 

.IRG BRENDLE 

ABSTRACT. \Ve show that it is consistent for the ahnost-disjointness lltnllber 
a to have couiltable cofinality. For exainple, it nlay be equal to 

INTRODUCTION 
Cardinal invarialltS of the COlltilllllllll, that is, cardimd mlmbers lmtxveen RI and 
c (the size of the COlltilllllllll) which atle defined a.s the smallest size of a familv of 
real nmnbers with a certain comhinatorial prolmrty, play an increasingly ilnportant 
role in moderl set theory. Equalities and inequalities between cardinal invariants 
have many commotions with problems arising lmturally in general topology, real 
analvsis and algebra, and, ri'oin a purely set theoretic point of view, there is a 
deep interplay with forcing theory, in particular iii the light of the search for new 
iteration techniques. 
One of the most basic questions al»out cardinal invariants is which values thev 
can assulne, and, for almost ail cardilmls, it is klown that anv regular value is 
possible, t Furthermore, most cardinals tan either be showll to be regular iii ZFC 
or they are equal to c in the l'andom real model, in the Çohen real lnodel. 2 or even 
in both, so that they can be consistently singular of uncoultable cofinality. Notable 
exceptions are the splitting llUlnber  of which it is still ullkllown whether it mav be 
singular [V] and the almost-disjointlmss lmmber a. which has recentlv been shown 
to be consistently singular of uncoultable cofinalitv bv Shelah [$2].  Things get 
trickier when one considers singular cardinals of countable cofinalitv. In fact, bv 
far inost of the cardinals, even those singular in the Cohen o1 randoln lnodels, can 
be shown to have uncountable cofinality in ZFC. 4 Exceptions are 
 Shelah [S1] has proved that the covering lmlnber of the mdl ideal may have 
countable cofilmlity, 
the ahnost -disjointness mmber a dealt with below. 
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 itis unknown whetller the reaping mmlber r [M. Problem 3.4] or the inde- 
pendence number i can have countable cofinality. 5 
Here we show 
Main Theorem..4.sume CH nd let Abe  singlr crdil of coq.table cofi- 
,ality. Then there is a forczg extension satisfyig a = A. In particular, a = R is 
consistent. 
 give a brief outline of the proof. Itis well known that, assuming C. one 
Call force a mad family of size R such that, by a standard isomorphism of names 
argument, there is no mad familv of size R for n k 2 in the generic extension [H1]. 
However, mad familles of size R may survive the forcing. The sinlplest wav to 
get rid of small mad familles is by iteratively adding dominating reals, sav for 
R2 steps. If this is donc in such a wav that every dominating real is dominating 
only over a fragment of the mad family of size Rw a¢l¢led in the initial step. the 
latter will survive. %t, if the dominating reals are added in a standard way. 
the t})rcing will lose most of its homogeneity, thc isomorphism of-names argument 
will ceae to work, and there may he a mad fimfily of size R2 instead. This is 
where Shelah's rcccnt technique of it«,ration along templates [$2] cornes in. 6 It 
provides for a. way of adjoining dominating reals with a forcing having enough local 
homogencity. However. since isomorphism-of names arguments require CH iii the 
ground model, we need to describe the two step extension sketched above in one 
step and incorporate the forcing adding the mad familv of size R into the template 
framework. This accounts for some of the tcchnical difficulties described below. 
Apart from proving the Main Theorem, we also present a new, more axiomatic. 
treatment of the teml)late framework (Section 1). While Shelah [$2] defines the 
teml)late via two procedures building more complicated sets (with larger depth) 
from simpler ones. we onh" require that the template is a family of subsets of 
the linear order underlying the iteration satisfying several axioms, most notably 
well-foundcdness.  think this apl)roach is more lucid, apart from providing for a 
simpler definition of the iteration. There is a price one bas to pay for that. however: 
the Completeness Lemma 1.1. showing that we are indeed dealing with an iteration, 
requires some additional work. Of course, our general approach can also be used 
to prove Shelah's original results [$2] (sec [Br]) and. siuce thcre is no need to add 
a mad family in this case, definit.ions will l»e simpler than in Section 1. 
In Section 2 we describe the tcmplate used for the proof of the Main Theo- 
rem, and in Section 3 we provide the isomorphism of nalnes argument needed to 
comp}ete the proof. 
The template framework developed in Section 1 for Hechler forcing can in fact be 
used to handle a large class of easily definable ccc forcing notions (see [Br, Section 
4] for a few examples). eplacing the forcing generically adjoining a mad fanfily 
bv ail appropriate relative, one can get analogous results for relatives of a. e.g., 
for as, the size of the smallest mad familv of partial flmctions from w to w. So, 
for example, as = Rw is consistent, and so is a = R < as = Rw. For the latter 

5As for maximal almost-disjoint families, it is rather easy to force a maximal independent 
family of size, say, R. Both r and i are equal to ¢. and thus possibly singular, in the Chen and 
random models. 
6As mentioned above, Shelah showed among other things that « could be singular. In his 
models, « is equal to ¢ and therefore has uncountable cofinality. 
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result, one needs to replace He«hler foreing by eventually-differeut-reals forcing in 
the framework of Section 1 (cf. [Br, Sectiou 4]). 
Let us briefly recall the maill notious relevant for this paper. Two infinite subsets 
A and B of w are called almost-disjoint if their iuterse«tiou is filfite. A ç [w]  is 
an almost-disjoin, t family if its lnemlmrs arc pairwise almost disjoint. A is a mad 
family (maximal almost-disjoint family) if it is lnaximal with respect to beiug an 
alluost-disjoint faufily, i.e., for everv B  [w]  there is A  A such that A  B 
is infinite. The almost-disjointness number a is the size of the least infinite mad 
fmnilv. For functious f,g  w , we say that g evedually dominates f (and write 
f * g) if the set {n; f(n) > g(n)} is finite. The unbounding nu,nber b is the 
cardinality of the smallcst unbounded falnily in thc structure (w , *), that is, the 
size of the smallest U ç w such that for all g G w  there is f G U with f * 9- 
The dominatin9 number 0 is the size of the least cofinal fitmily in (w , *). It is 
well known and easv to see that b  0 aml b  a iu ZFC [vD]. 
He«hler for«i,,9 [H2] D (see also [B.I]) «onsists of pairs (s. f) where s G w <, 
f G w w and s ç f, ordered 1,y (t,g)  (s, f) if t  s, and g k f everywhere. 
It generically adds a domin.ati 9 real, that is, a real that eventuallv dominates all 
ground model reals. It is this forcing adjoining a dominating real which we shall 
use in the template fi'aluework sket«hed al»ove. 
Oto" notation is standard. For cardinal invariants of the continuum, we refcr 
to [vD], [V] or [B1]. For forcing theory, in particular for forcing related to cardinal 
invariants of the contimmm, see [BJ]. 
I thank .huis Steprgns fi»r pointing out a flag iu an earlier version of this work. 

1. TEMPLATES AND ITERATIONS 

The lnOSt useflfi definition of a template seems to be (see also [Br]) the following. 

Definition (Template). A template is a pair (L,Z) such that (L, _<) is a linear 
order and 2- C_ 7)(L) is a family of subsets of L satisfying 
(1) {3, L e Z, 
(2) 2- is closed under finite unions and intersections, 
(3) if y < x beloug to L, then there is A G 2- n 7(L) such that y G A. 
(4) ifAG2-andxGL\A, thenAnLG2-, 
(5) 2- is well-founded, i.e., there is a fuuction Dp = Dp/" 2- -- On, called depth. 
recursively defincd by Dp(0) = 0 and Dp(A) = sup{Dp(B) + 1: B G 2- and 
/3cA} forA2-\{{3}. 
Here, Lx = {y  L; y < x} is the restriction of L to x. IrA C_ L, we define 
ZIA = {B ç A: B  Z}, the trace of 2- on A. (A, ZIA ) is also a template. 

Since 2- is closed uuder finite intersectious, if A  Z, theu ZIA = {B  Z: B C 
A}. 
In our context, we need to slightly revise this definition because we will hot 
only have "iteration coordiuates" LHech used for adjoining Hechler generics but 
also "product coordinates" Lmad used for adding a mad family. Siuce the former 
should be generic over some, but hot all, of the latter, we need to incorporate them 
into the saine telnplate fi'alnework L, and some of the above clauses should be true 
for all of L, while others need to be satisfied only for members of LHech- 
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Accordingly, let LHech and Lmad be disjoint sets, put L = Lnech U Lmad, and 
assume L is equipped with a. linear order. Further suppose 27 _C (L) satisfies, in 
addition to (1) and (2) above, the followillg clauses: 
(3') if .r G LHech and g G Lx, then there is A G Z  P(L) Slch that g G A; 
(4') if A 6 Z and .r 6 LHech  .4, then 4  L 6 Z; 
(5 ) the trace Z[LHech = {A  LHech; A G Z} is well-founded. 
as well as 
(6) if A G Z and .r G A, then L  Lma d ç A. 
This is the definition of "telnlflate" we shall work with for the relnainder of the 
imper. Notice that (5 ) lllea.llS in tmlticular that the depth functioli Dp depends 
only on the LHech-part, i.e., Dp(A) = 0 iff A ç Lma d and, recursively. Dp(A) = 
sup{Dl)(B) + 1: B G Z and B  LHec h C A  LHech}. 
(6) is a closure condition for the L,,,aa-parl which is needed to make the proof 
of glain Lelmna 1.1 l»clow go thlough, glore gclel'ally, we say that A ç L is «losed 
if A satisfies (6). (Se» Z consists «mly «»f cl»sed sets.) r arbitrary 4 Ç L, we then 
(Icfin« it.s «lo.ure c'(A) by c,(A) = A U .en (L  Lmad)- Thus c'(A) is the slnallest 
closed sel conla.ining A and al(ci(A)) = d(.4). 
The basic idca fiw lhc following, attelnpted, dcfiniti«m con,es from [$2]. It is 
modifie(l, however, due te our axiolnatic treatment of the concept of "template'" 
(see also [Br]) and because of thc inclusion of Lm. 
Definition (Iterating Hcchlcr forcing and adding a lnad falnilv along a telnplate). 
Assulne (L,Z) is as above. X> define, for 4  Z, bv recursion on Dp(A). the 
partial ordcr (p.o.) P[A (more explicitly, we define P[(A. Z). but we shall drop the 
reference to Z in case there is no ambiguity). 
 Dp(A) = 0. This mcans A ç Lm- P[A consists of all finite partial 
fulctions p with domain containcd in 4 and such that p(z)  2  for all 
:  doln(p) for Solne u = tv  w. The ordering on P[A is given by: 
q et.4 p if doln(q)  doln(p) a.nd 
- ,tq  , p(z) ç q(z) for all z  dom(p), I{z  doln(p); q(z)(i) = 1}1  
1 for all i  n q . 
 Dp(A) > 0. PI4 consists of all finite partial flmctions p with domain 
contained in A and such that 
- therc is t = tv  w with p(z)  2  for all z  doln(p)  Lmad; 
- letting .r = lnaX(dom(p) Lh). there is B  Z  P(A  L) (so 
 sP 
Dp(B) < Dp(A)) su«h that p[(A  L)  P[B and p(«) (,f) 
where s G w < and f is a P[B-name for an elelnent of w, such that 
p(A  ) ,eu ". ç " (thi mlS (.) i a et-m r for a 
condition in Hechler forcing ). 
The ordcring on P[A is given by: q etn P if dom(q)  doln(p) and 
- nq  n , p(z) ç q(z) for all z  doln(p) Lm. {z  dom(p) 
Lm; q(z)(i) = 1}  1 for all i G nq  " (this guarantees that the 
reals added in COoldinates froln Lma d are characteristic fulmtions of 
an allnost-disjoint falnily),  well  

7Note. however, that the first coordinate sPx of the condition p(x) is not a name. 
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- either y = max(dom(q) ç LHech) > d:, - max(dom(p) ç LHech ) and there 
is B E Z ç) P(A  L) such that p[(A  L),q(A  L)  P[B and 
q[(A  Lu) IB p[(A  L), 
- or x = max(dom(q)  LHech) = max(dom(p)  LH¢ch) and there is 
B  Z(A  L) such that p[(A  L),q[(A 
L) Net pi(,4  Læ), ],f are [B nalnes, 
L) Iel "](n)  y(n) for all ,'" (the last two clauses mean that 
q[(A  L)Iet "q(x)  p(')"). 
 bave hot argued yet that this recursive definition works at all. Thc point 
is this requires that all [A's be transitive, which is hot trivial because the sers 
B G Z witlmssing that q  p lnav dcpcnd on the pair (p, q). Therefore, to prove 
transitivity, we need to show that the [B COlnpletely embed one into the other. 
This will be donc in Main Lelnlna 1.1 below. 
Note that, once this is achieved, [(A,Z) = [(A. Z[A) is imlncdiate for A 
Of course, the above re«ursion also defilleS [A = [(A,Z[A) for arbitrary A ç L. 
If A. B G Z, A C B, thon [A C [B is immcdiatc from thc definition (becausc 
Z[A C Z[B in this case). This is lnuch lcss «lear if one of A or B does hot »elong 
to Z. Neither is it clear whether [A <o [B, the lnost basiç prol)erty thc al»ove 
recursive dcfilition must satisfy to nmke it an iteration, ex-en in case ln»th A and 
B corne from Z. This issue is addressed bv the following crucial lelmna. 
Main Lenmm 1.1 (Colnpleteness of elnbeddilgs). Let B  Z and A C B be 
closed. Then [B is a partial order, [A C [B and even [4 <o [B. More 
explicitly, any p  [B bas a cammical redu«tion P0 = Po(P, A. B)  [A such that 
(i) dom(p0) = dom(p)  A 
(il) s ° = s for all x G doIn(p0)LHech and po(x) = p(x) for all x G doln(p0) 
Lmd (in particular, po = p ), 
and such that, whenever D  Z. B,Cç D, C close& C eends Po, then there is q G [D exte'nding both qo a,d p. 
Note that we do not require p B P0- 
Pro@ By recursion induction on , silnultaneouslv for a.ll templates (L,Z), 
 we prove that [B is indeed a p.o. (i.e., transitivity holds) for all B 
with Dp(B) = a; 
 we prove [A C [B for all B G Z with Dp(B) = a and all closed A C B: 
 we construct P0 = Po(p,A, B) satis[ving (i) and (il) for ail B G Z with 
Dp(B) = , all closed A C B and all p  [B: 
 we prove that for all B,D  Z, B,C Ç D, C closed, with Dp(D) = 
and all p G [B, letting A = C B and P0 = po(P, A. B) (which has been 
constructed either at stage a or at an cmlier stage), there is q  [D as 
required. 
Notice that if Dp(B) = a, D = B and C = A. then the latter indeed shows 
[A <o [B. 
The case a = 0 is trivial. So assulne  > 0 and Dp(B) = a.  first check 
transitivity of [B. Assume that r etB q etB P- Then clearly doln(r)  doIn(p), 
n r  n p, p(z) ç r(z) for ail z G doln(p)Lmaa, and I{z G doln(p)Lmaa; r(z)(i) = 
1}l  1 for all i G n r  n p. Let z = nmx(dom(r)  L«ch), y = max(dom(q) 
LHech) and x = max(dom(p) LHh). Then there are A0, A G Z  P(B  L,) 
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such that pI(B ç Lz),q[(B ç Lz)  P[Ao, qI(B c3 L) -<et,% pI(B G L), and 
is a P[Al-name and. in case 9 = z, ¢g is both a [A-nalne and a PA0-name 
as well as, in case z = 9 = z,  is a PA0-name. Let 4 = 40 U Al. Then 
A Ç ZGç(BL) so that Dp(A) < Dp(B), and we know bv the induction hypothesis 
that PAi <o P[A for i = 0.1. Therefore, p(BGL),q[(BGL),r(BGL) 
and r[(B  L) PrA q[(B  L) PrA p[(B  L). Moreover,  is a P[A nalue 
and, in case y = z, ¢g is a PrA llalnP and r[(  L)IbtA fg   as well , in 
ce z = 9 = z,  is a Pt.4 llalllC glld q[(B  )Ikr f  g. Taken together. 
this shows that r tB P as requiled. 
Now let A C B  Z, A closed, bc given. Assmnc r  PA. Let z = lnax(doln(r)G 
LHech)- Bv definition of the iteration thcl'e is A  (Z[A)  (Lz) such that r[(A 
L)  Pli and  is a P[A naine. There is   Z[B ç Z such that 
Clcarly z  . Bv clause (4') in the dcfinition of a tcmplate, we may therefore 
assume without loss of generality that B ç Lx. Thus  C B and Dp() < 
Dp(B) = a. By induction hypothesis, PI,4 C P and [ <o P[. Therefore, 
'Ç is a P[ naine as well. r G P[B follows imluediately. Hence PI 4 C 
required. 
Next assume also p  PIN is given.  construct P0 = Po(p.-4, N). Put z = 
max(doln(p)  LHech)- By definition of the iteration there is   Z  ç(B  L) 
suchthat=p[(BL) F[and jisaF naine. Put A= A. Note 
that   Z[A. Bv indu«tiol hypothesis, ç has a reduction 0 = P0(ç, -, 
sat.is-ing the barred version of the clauses of the lnain lelnlna. The definition of 
p0 split.s into t.wo cases. 
Case 1. z  4. Then dom(p0) = doln(p)  .4. p0[(A  L) = 0 and po(z) = p(z) 
for z Ç (dom(p)  A)  Lx. (Note that such z must belong to Lmad-) 
Case 2. z e A. Then let doln(p0) = doln(p) A. P0 t(-4  L) = 0 and P0 (z) = p(z) 
for z  (dom(p)  A  Lmad)  L x.  know by induction hypothesis that PA 
<o [. Therefore. there exists a caomcal projectiott o to . of the F- 
naine . Accordingly we let p0(c) = ,,(P 
More explicitly, we do the following. For silnplicity work with the cBa's A = 
r.o.([.) and  = r.o.([) associated with . and .  know by the induction 
hypothesis that A <o . Note that   s ç . In , for all 
with s ç s, we let b s ç . So b = pand. for n > ] , the b, 
Isl = n, are a maximal antichain below . Let a be the product (intersection) of 
0 and the projectio of b t.o A (recall the projection of bs to A is the unique 
condit.ion a such that a OE b and any extension of a in  is colnpatible with b). I11 
particular, a = 0 and {a; I1 = ) = 0 for  > I1. Defin  by recursion 
on n = ]s] as follows, a = a = 0 and, for n > I1, st,  = ¢_. 
j<s(-l) a* 
t(_)-(j)) (which is equal to at(_ )  (a  j<(_) at(_F(j)) ). 
Then one can show by induction on  > Isl that the a, Isl = ,, are a lnaxilual 
antichain below P0- Therefore they canonically defiue a F-name o (that is, 
 s ç " 
' = {a, s' < s evervwhere, 
The main property of this naine is t.hat for ail s, a ; _ 
 ç , I'1 = I1) is a reduction (not necessarily the projection) of bs = 
 everywhr«  ç 1'1 = I1}. (This is so because (a)' = {«-a* s' 
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everywhere, sP C_ s, 18'1 = 181} is the prod,,ct of i#0 and t.he projection of b' s alld, by 
, '  (aS)' is trivial.) 
the definition of as a s 
This completes t.he defilfition of P0. Clauses (i) and (ii) are triviallv satisfied in 
each of the two cases. 
Now aSSulne B. D  Z, B. Cç D, Cclosed, are such that Dp(D) = 
p  P[B and P0 = P0(P, A, B). Let q0 tc P0, qo  PIC.  need to constru«t q. 
X = lnax(doln(p)  LHech), , A,  and P0 are as in the previous construction. 
Case 1. z  A. So x  C. Let g = lnax{z < x; z G dom(q0) 
can find Ê e Z[C  P(Lu) such that qo[(C  Lv) e F[Ê and  is a F[Ê naine. 
Thereis   Z[D ç Zsuch that Ê = C. Since.q  Ê and.q 
follows. By clause (4 ) in the (tefinition of a telnplate, without loss of generality, 
 ç L u. By clause (3') ill the defilfition of a telnplate filld  G (Z[D)(Lx) 
contailfing g. Let  =  U  U B ç L and ç = (  C) U Ê u , Ç L» By 
clause (2) in the defilfition of a template,  G Z[D ç Z and  G Z[C. Since 
Dpzrc(Ê)  Dt)z)c(Ç)  Dpz(D) < o, O,) = qo [(C  Lz)  P[OE 1)v the inductiol 
hypothesis. 0o O Po and Ç = . are ilmnediate. By the inductive assmnption 
for the barred version, there is q  P[ extending both O0 and . 
 define q such that 
 dom(q) = doln(q) U doln(p) U dom(q0), n q = 
 q(9) = q0(9) for all 9 ¢ (dom(q0) N Lz)  Lnch, 
 q(x) = p(x), 
 qo(z) ç q(z), q(z)(i) =0, for z  (dom(q0)  L)L.ma and i  nq n q°, 
 p(z) ç q(z), q(z)(i) = 0. for z  (dom(p)  (Lz U clora(q0))) Lma and 
It is straightforward to check Oint q  [D aud q ev q0- So let us al'gue that 
q ev p as well. Clearly n q  n p.  need to show that p(z) ç q(z) for ail 
z  dom(p)  Lma. This is obvious for z < z be«ause ç e . It is immediate 
by definition for z > x belonging to dom(p)  dom(q0). So assume z > x, z 
dom(p)  clora(q0). Then p(z) = po(z) ç qo(z) ç q(z), as required. Next fix 
i  n q  P. We need to check that there is at most one z  dom(p)  Lma with 
q(z)(i) = 1. By way of contradiction assume this is true for two distinct z0 < z. 
By construction we nmst have i  n q°, x < Zl and Zl  dom(q0)  dom(p). Hen«e 
Zl  A. Therefore z0 nmst belong to A as well because A is closed. Thus both z0 
final Z 1 belong to clora(p0). Tlfis means that q(z)(i) = qo(z)() 
whi«h «ontradicts q0 etc P0, and we are done. 
Case 2. x  A. So   C. Find   Z[CP(L) such that ç0 = qo[(CL) 
and o is a [-name. Without loss of generality, , ç C. C =  is immediate. 
There is   Z[D ç Z such that ç =   C. Sin«e x  , we get x  . Bv 
(4'), without loss of generality,  ç L. Ve may also qsume  ç . Since 
Dpz( ) < Dpz(D ) = a, we can fi'eely use the induction hypothesis when dealing 
with A, , C, and . In parti«ular, ç0 ee P0- 
Now note tn,t we have s = s  ç s  and 0 Iee s  ç f. Let m = I%1 . 
Since also 0 Iee "ëq  >_ ] (everywhere)", we see that q0 Iete 
' ff m < s eA P where we let p = 
Hence we get a := a qo = 
(the canonical reduction of ç0 to F[: note here that Dp(Ç)  Dp()  , so 
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that/ indeed has been defined already). However. bv construction, a <-'tA/30 is 
nothing but a reduction of b := b',o = []m G s° p G  to . So there is 
ri+  P[ such that + 5t ç and ri+ t b (so that, in particular, 
and ç+ Ikt  ¢tm  s). Let  = p0(ç+,A. B) be the canonical reduction of 
ri+ to P. Then  eb ç;" Therefi»re ç and 00 have a common extension 
ï in D[Ç. Bv inductive assmnption for the bm'red +-version. there is 
extending both + and 
XX define q such that 
0 + 
 dom(q) = dom(0 +) U dom(p) U dom(q0), v= n 
 q() = qo() for all 9  dom(q0)  LUe«h xvith 9 > x 
 s = s  and . is a P[ naine sud, that 0 + lket  = max{¢, f}, 
 q,,(z) Ç q(z), q(z)(i) = 0, for z e (clora(q0)  Lx)  Lmaa and i e ,,q  n q°, 
 p(z) Ç q(a), q(z)(i) = 0, for z e (dom(p)  (L U dom(qo))) Lmaa and 
i  q k P- 
To sec tiret q  [D. note that + ItD s  ç J: by construction. It is then 
sraightfi»rward  check that q tD qu.P- In fa«t, for q ?tD P we argue  in 
Case 1 al,ove. 
Note that. as an immediate consequence of lain Lemma 1.1, we get that for 
arbitrary closed A ç B ç L, F[(A. Z[A) completely embeds into F[(B. Z[B). 
Lemma 1.2 (Chain condition). Let .4  . Ang mcomdable K Ç F[A bas an 
un«oudable certered subsct. 
Pro@ Bv a standard A system argument, it suces to show that 
n p = t. sç = s for all x  dom(p)  dom(q)  Lunch, and p(x) = q(x) for all 
z  dom(p)  dom(q)  Lmad. then there is a common extension r with dom(r) = 
dom(p) U dom(q), n  = n  = n . 

if x  dom(p) C/LHech, 
if x C dom(q) çl LHech 

raid 
r(«) = { p(.r) if « C dom(p) ç/Lnd, 
q(x) if x  dom(q) ç/Lma d. 
We do this bv induction on Dp(A). 
The case Dp(A) = 0 is trivial. So assume DI)(A) > 0. First assmne x 
mmx(dom(p)  Lch) < y = max(dom(q)  LHch). Then there is B  Z P(A  L) 
such that pL, q[L u  PB. and  is a P[B naine. By the induction hypothesis, 
ve get the required fiL u %et pL.q[L. Let r(y) = q(y) and let r(z) = p(:) for 
z e dom(p)Lmad, z > y. and r(z) = q(z) for z e (dom(q)  dom(p)) Lmad, 
Next assmne x = max(dom(p)  LHech) = max(dom(q)  LHch). Again there is 
B e Z  P(A  Lx) such that p ILs. q tLx e P tB, and ]. ] are P tB-names. Again 
we get rtL«. Let s; = s = s and ]" be such that r[L It ]" = max{j,]}. 
Also let r(z) = p(z) for z e dom(p) Lmd, z > g. and r(z) = q(z) for z 
(doln(q) k doln(p))  Lmad, Z > g. 
Lemma 1.3 (Embedding Hechler forcing). Let x  LH«h and A 
T5en t5e two-step iteration FA .  that canonically adds a Hec5ler-generic 
coordinate 3" over te generic extension via F[A contpletely embeds into FL. 
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Pro@ Let B = cl(A U {x}). I?[B embeds into P[L bv Main Lemlna 1.1. So it 
suftïces to show P[A * b <o F[B. This does hot follow from (file statelnent of) 
Lenmla 1.1 because A U {x} need hot be closed, but it is relatively straightforward 
from the proof of 1.1. 
More explicitly, given p  [B, there is   Z[B  (L) such that fi = pILe 
P[ and ] is a P[ llalllç. Without loss of generality, ,4 ç . By 1.1, P[A 
o P[B. Therefol-e, p h a CmlOlfical l'eduction ç0  [A. As in Case 2 of file proof 
of 1.1. there is a canonical projection ]Ço to P[A ,f ]. Define l'o e L4 *  by 
po[A = P0 and p0(z) = (s, ]o). As in Case 2 of the proof of 1.1. argue that any 
q0  P[A * b extemting ivo is compatible with p. 
This may badly rail in case A  Z because then [A * b lmed hot elnbed into 
Lemma 1.4 (Names fi)r reats). Assume iv  [L avd ¢ es a [L ,ame for a real. 
Then there is A Ç L tout, table such lhal. Icltin 9 B = ci(1), iv  [B, ad ] is a 
P[ B vaine. 
Pro@ The proof proceeds bv sinmltaneous inductiol on Dp(L).  ithout loss of 
gelmrality. Dp(L) > 0. 
Assume first p  [L. Let x = max(dom(p)  LHech)- There is C  Z  
such that p[Lx  [C and ] is a [C-lmme. Bv ilduction hypothesis, there is 
A0 ç C COUltable such that pile  [c1(,4) and ]ç is a. [cl(A0) lmme. Then 
p  [B where B = ci(A), A = Ao U dom(p). 
Assume llOW that ] is a [L naine. By cCC-lmSS (Lemma 1.2), there are 
i, n  } Ç [L and {kn,i  : i. n  } such that 
 {p,i; i  } is a nlaximal alltichail in [L for all n  w. 
By the previous paragraph, we can find countal)le sers A,i such that p,i 
[cl(A,.i). Put A = i. 4,.i, B = ci(A). Since [B <o [L (Lelmna 1.1). 
we can construe ] as a [B-nalne. 

Assume (L.Z) and (L. `7) are telnplates and Z C_ `7. \Ve sav that Z is cofinal in 
`7 if for all x 6 LHech and all _4 6 `7 A/)(L:) there is t3  Zç7a(Lx) containing ,4. 
The following is. iii a sense, a trivialitv. 

Lemma 1.5 (Cofinal subtenlplates). If Z is «ofirml in . then P[(L,Z) is forcing 
equivalent to P['(L,,7). 

Pro@ By induction on Dp(L) (in the sense of 2-), we argue that conditions in 
PI(L,:/) and conditions in PI(L,) can be canonicalh identified so as to vield 
forcing equivalence. Without loss of generality, Dp(L) > O. 
Let p  PI(L,). Put x = max(doln(p) ç? LHech). There is A   C P(L,) 
such that p = p[Lx ¢ P[(A.,7) alld ]P is a P[(A. ,,ï)-llallle. Since 7 is cofinal iii 
,7, there is B  2- ç/)(L) such that A _C B. Bv Main Lemlna 1.1. we know that 
PI(A, `7) <o ]PI(B, ,7) and, by the induction hypothesis, PI(B, .7) and P[(B.Z) are 
foI'cing equivalent. Therefore, we lllay construe /3 as a condition in P['(B,Z) and 
ff as a PI(B, Z) naine. Thus iv ¢ P[(L. Z). It is straightforward to verifv that this 
identification induces forcing eqnivalence. [] 
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Proposition 1.6 (Adjoining a scale). Assume tt is regular uncountable, p C_ LHech 
is cofinal izt L, and La  Z for all c < p. Then I?[L forces b =  = p (i.e., there 
is a p scale). 

Pro@ For each a < t I, let J: be the lmme for the Hechler-generic adjoined in 
coordinate a of the iteration (see Lemma 1.3). By construction, the fa are forced 
te be wellordered by <_*. Let ) be a I?IL-name for a reM. Bv Lemma 1.4, there 
is A C_ L counta.lle such that ) is a I?[cl(A)-name. Since t is regular uncountable 
and cofinal in L, there is a < tl such that ci(A) _C La. Since La  27, fa is forced te 
dominate the reals i the generic extension via II[La and, a fortiori, it will dominate 
.¢ [] 

Proposition 1.7 (Adjoining a mad family). Assume L bas uncounlable cofinalitl 
and Lmad i8 c&nal it L. Then FIL cano,tically adjoins a mad family of size [Lmadl- 
Pro@ Let G be F[L-gcncric over the ground model. For x G Lmad define t = 
{tf e ; p(x)(t,) = 1 for seine p e G}. Let M = {};" .r e Lmad}- By definition of 
the p.o., M is an ahnost-disjoint fanfily. We need te check maximality. Se let ) be 
a FIL naine for an infinite mil)set of w and assmne bv way of contradiction that 
p forces that 2 is ahnost disjoint frein ail f. By Lemma 1.4 there is a countable 
set A such that p  F[d(A) and 2 is a F[d(A)-name. Since L has mwountable 
cofinality and Lmd is cofinal in L, thcre is " G Lmad such that al(A) Ç L» By 
Main Lemma 1.I we know that P[cl(A) <o P[L <o P[L. 
Find k0 and P0 etz P such that 
Put P0 = po[L. Clearly any , 9  dom(0)  Lmd, is a PL-name. Se we tan 
find k k k0 and Pi tz 0 such that 
for all .q  dom(ç0)  Lmd- Since 2 is forced te be infinite, we cau find 
and i0 k k such that p Ietz, i0  2. Without loss of generalitv, n  > i0. Then 
we nmst necessarily have (9)(io) = 0 for ail .q  dom(p0)  Lmd. 
Define a condition p by 
* dom(p) = dom(ç) U dom(p0), 
* p(z) = po(z) for ail z 6 LH«h 
* p(z) D p0(z), p(z)(i) = 0 for all i with 
Lmad  dom(p0), z > , 
{ 1 ifi=i0, 
p(z)(i) = 0 for all i 
It is straightforward te verify that p  P[L and that p Nelz P0. Since 

we have a contradiction. [] 
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. BU1LDING A TEMPLATE FOR ADJOIN1NG A MAD FAMILY 
For simplicity asstlme CH for the relnainder of the pat)er. 
Assulne A0 k R is regular, and A > A0 is a singular cardinal of countable 
cofinality, say A = Un An, the Anbeing regular, equal to A n , and strictly increasing. 
Also sui)pose no < A for n < A. As usual, It* denotes (a disjoint copy of) 
with the reverse ordering. Elelnents of i t will be called positive, and lnembers of 
are e9ative. For each n choose a 1)al'titiol A, = Ua<w, s,î such that each S is 
co-initial in ,. Also aSSUlne S,  A = S,c for m < n. 
The following definition is lnotivated by Shelah's work [S2]. 
Definition (Template for adjoining a mari falnily). Define L = L(A) as follows. 
Elements of L m'e lOnelnpty finite sequen«es .r (i.e., dom(z) Ç w) such that 
 z(O)  
 x(n) Ç ; U , for 0 < ,t < Ixl- 1, and 
 in case Ixl  2, if .r(Ixl- 2) is positive, then x(Ixl- 1) Ç AleI_ 1 u A, and if 
.411 - ) i negative, then .r(I.r I - l) Ç A* U 
Sgy .F e LHech if Ixl =  or (11 - ) e * 
Ixl-1 U lxl-l" Otherwise .r Ç Lmad- (This 
lneans that x e Lmd iff Ixl 2 oe a.d either x(x-2) is 1,ositive and x(Ix- 1) 
or .r([x- 2) is negative and '(Ixl- 1) < * 
- Ixl-1-) Equip L with the following 
lexicographic-like ordering: .r < y iff 
 either .rC y and (Ixl) i positive, 
 or  c . lld x(lyl)is negative, 
 or. letting ,, := nfin{m  .r(m)  y(m)}, either .r(n) ia negative and y(n) is 
positive, or both are positive and x(n) < y(n), or both are negative and 
.r(,) <a. y(n) (i.e., there are a < fl <  such that x(**) = ff* <a. a* = 
()). 
It is immediate that this is indeed a linear ordering. 
We identify sequences of length one with their ranges so that 0 is a cofinal 
subset of L. Sav x Ç LHech is relevant if Ixl 2 3 is odd..r(n) is negative for 
odd n and positive for even n, x(xI- 1) < w, and whenever n < m are even 
such that x(n),a'(m) < 1, then there are fl < a such that .r(n- 1) Ç S_ 1 
and x(m- 1) e S_. For relevant x, oet & = [t(Ixl- 1),x), the interval of 
nodes between .ri(Ix [ - 1) and x in the order of L. Notice that if x < y are 
relevant, then either ,J  Ju =  or & C ,lu (in which case we also bave Il  Ixl, 
:F(lyl- 1) = yF(M- 1) and .r(ly]- 1)  y(lyl- 1)). 
Define Z = Z() to be the collection of all finite unions of sets of the fornl 
 LforaGA0, 
 ci(&) for relevant x, 
 cl({x}) for x Ç LH«h, and 
 L  Lad for x Ç LHch. 
So L() is a subtree of (* U) < (i.e., it is closed under taking iifitial segments). 
The nodes belonging to Lmad are exactly the terlnilml (= maximal) nodes of this 
tree. The point of the Jx is that we need "copies" of the large supports given by 
the L for isomorphismf-names arguInents. The S, then, are used to code the 
places where we put the .J so that we basically get well-foundedness for free. 

Lemma 2.1. (L,Z) is a ternplate. 
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Proof. Clauses (1) and (6) in the definition of template are immediate, as is closure 
undcr finite rotions. To see closure under finite iutersections, it suffices to argue 
that the intersectiou of any two sets of the above form (i.e., Le, ci(Je), d({x}), 
and L  Ld) is again of this form. This, however, is straightforward so that (2) 
holds as well. 
To prove (3'), lct x G L«h and g G L,. In case g G LH«h, we have g G c({g}) 
For (4), it suffices again to consider sets A from Z of the above form. Let 
x G L«h 4. Without loss ofgencrality. AL« ¢ 0. If 4 is ofthe form L 
d(Ju), cl({g}) or L u  Ld, then we must have V > x. A = L u is impossible and 
if A = cl(Ju), then x < g[(]g]- 1) = min(Ju). So, in each of the possible cases, the 
intersection with L, is L,  Lmd- 
 are left with showing well-foundedness (5). Assume A,,., n G , is a de- 
crcasing chain ri'oto Z[LH«h. Let a be such that Le,  LH«h occurs in A, as a 
component. Choose a, o milfimal among thc a. Without loss of generality, n0 = 0. 
Then all Le,  Lch are thc saine and it suffices to consider the J«-components. 
Thus we may assume, without loss of generality, that A0 =   LH«h, and there 
is a finitely i)ran«hing tree T ç w < such that A,, = («¢Tm- "]x U Fn)  LHe«h 
where the #Ç ç LHch are finite, and such that a Ç , la I =  < I1 = m. implies 
.1 Ç ,1, and such that the .1«, a G T  , are pairwise disjoint. Now note 
tlmt if f G [T] is a bra.nch, t/mn the sequence {.rI"; ,, G w} must eventually sta- 
n} wouhI constitute a decreasing sequence of ordinals. Then notice that if Ix{ 
is eventually constant, so is the decreasing sequence x{I'(]x I] - 1).) Since T is a 
finitely branching tree this means that the total number of the x is finite which 
in turn entails that the sequence of the .4, eventuallv stabilizes. 
Corollary 2.2 (Bounds for a). FIL forces b = 0 = Ao and adjoins a mad familv 
of size  (so that o  a   ). 
Proof. This is immediate bv construction of FIL and Propositions 1.6 and 1.7. 

3. KILLING MAD FAMILLES USING TEMPLATES 
We are left with showing there is no mad familv of size legs than A in the 
generic extension. As explained in the Introduction. this is an (albeit sophisticated) 
isomorphism of-names argument. Isomorphisms of names canonicallv boil down to 
certain brands of partial isomorphisms between subsets of L. and we begin with 
their investigation. 
Definition (Isomorphism). Let A.B C_ L be countable trees, s Call A and B 
isomorphic (.4  B) iff there is a bijection ¢ = CA,U" A --, B such that 
(a) I(«)1 = I1, 
(b) 4(«)In = 
(c) « < y iff re(x) < re(g), 
(d) x(n) is positive iff ¢(x)(n) is positive, 
(e) x G Ld iff ¢(x) G Ld 
for all x, g G A and all  G w, and such that 

8F{ecall A is a tree if it is closed under taking initial segments, i.e., given x Ç A. we have 
x[nÇ,4 for ail n Çw. 



THE ALMOST-DISJOINTNESS NUMBER 2645 

(f) Z[A is nml)pcd to TIB via 
Since the trace of Z on each comltable set is comltable, there are at most 2 ° = 
isoinorphism types. 
This, the strongest notion of "isomorphism" we shall consider, vill be used in 
several pruning arguments below. However. tbr most purposes the following is 
sutiicient. 

Definition (Veak isomorphism). Let A./3 C_ L be a.r|fitrary. \Ve sav that .4 and 
/3 are weakly isomorphic (.4 wk /3) if (e) is satisfied and instead of clauses (c), 
(f) we have 
(c') x < y iff ff(x) < ff(y) fol" all .r.y sueh that there is z G LHhçA with 
x<_z<_y, 
alld 
(ff) ff lllaps a cofinal SIlbSCt of Z[A to a cofilla.1 subsct of 
respectively. 
Lemma 3.1. Let 4 md t3 b« countable trees such that LHch ç A (L«ch  t3, 
respectiv«l.q) i,s cofinal i, A (in t3, resp.). If A - 13. as witnessed by ff), then there 
is l, ectending ff and wit.nessing that ci(A) wk ci(B). 
Pro@ Call a nonempty X C ci(A) V1 Lmad counected if given x < y ffoto X, the 
interval Ix, y] is disjoint ffoto A ç LHch. A maximal coimected set is called a 
connected comportent. Note every commcted COlnponent has size A (because LHech 
is cofilml in 4) and ci(A) ç? L,nad is a disjoint union of at lnost comtablv man.v 
connected components. 
Given x  LHechç?A with .r([xl-1) being positive, put Comp. r 
z for every z  A ç? LHech with z D x and y > z for every z  A ç? LHech such that 
z[[x I < x}. Clearly Comp« is a commcted colnponent. Duall.v, define Comp.,. for 
x  LHech ç? A with negative x([z I - 1) by interchanging < and >. 
For each y  ci(A) ç Lmad. there is x  LHech f-I A with y  Comp. To see this, 
let n < [y[ be lnaxilnal such that y[n  A. Assume. without loss of generality, 
is positive. Let k <_ '« be minimal such that all y(i) for k _< i _< n are positive. 
If possible choose m, k _< m < ,, and x  A çl LHech , Ix[ ; .l ÷ 1, such that 
xlrn = ylm, x(m) > y(m) is lninimal, and such that m is the maxinml value for 
which such an x can be round. Then y  Comp«. If m and x camlot be round, we 
let x = ylk and check y  Comp.Æ (note that (Ixl- 1) ; .v(- 1) is negative in this 
case so that the second alternative of the definition of Comp applies). Therefore 
ci(A) Ç1 L,nad = UzLHechA Cmp.. 
Also notice that for x.x   LHech çl A, if Comp« = Comp,, then Comp¢(x ) = 
Comp¢(x,), and if Comp« ç Comp, = 0, then Comp¢(x 
So we can simply extend ff to g, by mapping Comp. to Comp¢(x ) for all x 
LHech  A. Then (c ) and (e) are ilnlnediate. To see (f), note that bv defiifition 
of the template, sers in Z[cI(A) that are Ulfions of sers ff'oto Z[A and of sers of the 
form L V1Lmad are cofinal in :/TIcl(A ). However, since ff identifies sers of ZIA and 
sets of Z[B. b identifies sers of the latter kind. [] 
Note that we did hot use the full st.rength of out notion of isolnorphism in the 
above proof. Clauses (c) and (f) could be replaced by (d) and (f) respectively. 
Furtherlnore, instead of dealing with tlees A and/3 (and having (a), (b), and (d)), 
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it suffices that ci(A) çl Lmad is the ullion of the COlni)onents Comp., x E LHech A A. 
and similarlv for/3, and that extending ¢ by lnapping Cmp. to Comp¢(z) preserves 
Lemma 3.2. If A w«k B. the,, PI,4  
Pro@ Notice that clauses (c), (e) a.nd (f) are enough to guarantee that 
D[B. Bv Lelmna 1.5, this is still true if (f) is replaced bv (g). Finally, bv the 
cas" PI,4 is defined recursively, interchalging elelnents of Lmad that belong to the 
saine COlmected COlnponent of 4  Lmad does hot affect the p.o. 9 (because the 
interchalging mai) sends a cofilm.l subset of Z[A to a cofinal subset of Z[A. see 1.5). 
Completion of ttte proof qf the kIaiz Tteorem. Now assume À is a naine for an 
ahnost-disjoint familv of size < A, say À is listed a.s {,4; a < h'}. Also assulne 
is forced to have size at least A0. Let k <  be maximal such that ,  Ai.. Without 
loss of generality, h  A-. 2. 2, shall perform several standard pruning argulnents, 
reordel'ing the family of the À' so that the first A many look very "silnilar", that 
is, those  that do llt fit the t)att«ll get relnoved to higher indices. This is whv 
we stilmla.te h  A- 2. Eventually. the first  lllallV Àa will suffice, and it is those 
that we use to create a lleW llallle ,h «itnessing non-lnaximality. 
For fixed a, fiud c«mntablc maximal anti«hains {P,i; i G } ç 
tlid {,i e 9. i, , e w'} Sllch that  4" 
Pmi lb   . iff .i = 1 and 
iffk.i = 0. Let B  = 0{dom(p.i): i.n  } ç L. B  is at most countable. 
Without loss of generality, it is a tïe. Let C  = cl(B). Put B :=  B . So 
 < + < A. Bv CH and the  systeln lemlna we inay SUlne, without loss of 
generality, that tho {; o < N} forln a -systm. ald that th bijction  = ' 
(see above) sending B fo B is an isolnorphisln fixing the roof R of the svstem. 
Beca.use t here are only Aï i =  - i mmv COllt able subset s of L  h  (* U A) . we 
may also assume that if' Lech and   '. then z  . Also stipulate that 
thre is SOlne 00 < 'i such that whenevr o < Nh,   , j odd md (j) 
then (j)  N for SOlne 0 < 00. As xplaind above,  cmonically inducs a 
isolnorphisln  = @' betwe«n C  and C  (Lemlna g.l), which in turn yiIds an 
isolnorphism ç = i a, bet.ween PC  md PIC  (Lelmna g.2) both of which elnbed 
iuto FL (Main Lelmna i.I), as well as between FC-imlnS ld [C-naluS. 
Furthernore, since connected components are holnogeneous froln the forcing point 
of view, since C   L«h = B  L«h is countable, and since C   Lmad h only 
countably inany COlmeCt.d components (see th proofs of Lemlnas 3.1 and g.2), it 
bas, up fo isomorphisln, only 2 ° = N inanv isolnorphisln types of nalneS. (Of 
course, there are a total of A ° nalneS.) Therefore, we lllV g]so suppose that 
identifies  with .. which in,ans, more xplietly. tlmt ,i := '.i = ' md 
p 
XçS'ite B  = {z: s  T} where T ç ( Ul) < is the canonical tree isolnorphic 
to anv B . This means in particular that '(z) = «, that I1 = I1, and 
that s(z) is positive iff z(n) is positive. Let THech = {$  T:   LHeeh} and 
Tmd = {s  T; z  Lm }. Tn«h is a subtree of T, while Tmd is a set of terlninal 
nodes of T. Furtherlnore, let S ç T be the subtree of T corresponding to the root, 

9That is, the order structure on connected components of .4f3 Lma d is irrelevant, and connected 
components are homogeneous from the forcing point of view. 
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a a xs  iff s E .9. Furthermore, 
that is, sE Siffx» G R for allo. So. foro ¢/3, m s = 
if s G THech  S. then [s I OE k + 1. List {t G T S; t[([t[- 1) G S} = {t.; n k 1}. 
For a < fl define 
{ mill{n; ei{her t.([t,,[- 1) G w and x a X  
F({,}) or t(l/l- 1)  w alld a'a - - 
= t.(If. I 1) <a-.(It I 1)} 
if such ail it exists, 
0 otlwrwise. 
Note tha.t for each n k 1, eVely subset of Ai. honlogeneOllS in color , lll/ISt be finite. 
Using partition calculus as well  standard pruning argtllnelltS, we nlay therefore 
assmne that for all a < Wl, if s  S and s{Ç)  S. then 
 if ç is positive, then 
-(ç)(s]) for all  < , and all x  
are larger than Wl, and if Ç < , s{Ç),s{)  S, then 
- either for ail a, fi, we have 
(this is t.he case when sup z(ç)(s) < sup ()(lsl)), 
-- or for ail ( < 
x.<>(Isl) <  <ç>(Isl) 
(this is tllc cse whPn sllpa 
 if Ç is llegative, then x-(ç)(sl) > x  
S'(Ç),S'{}  ,ç, t]len 
- either for ail a, 
- or for ail a < , we have x a 
Define x s  L by recursion on the length of s  T  follows. If s  S, then x s = x s 
(a fortiori Ix;] = Ixl = Isl). If s  S lld s'<0  Ha  S, 1Pt x-«>(l*l) be th 
linfit of the x-<ç>(ll) (so it is either the sup or the inf, depending on whether Ç is 
. 0o 
positive or negative). Next find % < ll+l,  > Wl and s  SII+I, such that for 
all such s and Ç, 
 if xç.<o(Isl) = sup.x.<o(Isl), thon for ail 
 if x2.<o(Isl) = iIlf. Xs(ç)(Isl), then for 
-«> r(Isl + 1), we hv y(Isl + 1) < %. 
It is clear that we Call find such s's because ls[+l > [BI is regular (since [si k k). 
To complete the definition of xs-(ç ) stipulate I s-/01 = Ix-<ç>l + o I*l + 3, and 
define 
{x 
,-(ç)([s[) if [si > 0, 
a--{ç)(Is I + 2) = ç if Isl = 0. 
If s  S and s'(ç)  Tmd S, find 7  A* OA such tiret ()  Lmd and 
for ail   B with [IsI = ., we have (Isl) # . Such  clearly exists because 
A > IBI . Stipulate Ia.-{ç)l = I .-{ç)l = Isl + 1 and let a.-{ç)(Isl) = 7- Finallv. for 
the l'emaining t  T, stipulate again laîl = laPl + 2 = Itl + 2. find s c t with s  S 
maximal, and ptt aî[(Is ] + a) = a and a'î( + 2) = m() for  > Is I. 
Let B  ={.'., s T}. Notice t, hat B , although verv trelike, is not a tree 
like the B's. We proceed to verify that (B.gB ) and (B.g[B), a < w. are 
weakly isomorphic (clearly (a), (b) and (d) will rail but this is hot relevant for us). 
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 a ' \Vithout loss of generalitv, o = O. (c') 
Fix o and define ¢ = ¢" by ¢(a ) = x. . 
and (e) are ilnmediate by construction. So let us check that the trace of  on B ° is 
lnapped to a cofilml subset of [B . First fix  and consider L. Notice that there 
.o  L 
is ff0 5  such that ç(L&B °) = L&B  = LB . For any s  T with 
 . (0) > 
vet x s ¢ LB one lnust have a (0) > /5  0 and x(0) is an oe-lilnit of 
the x(0)'s. In particular, for ail such s. x(0) nmst have the saine value, say 70- 
Also .r(1) = 70 and «(2) = s(0) < w. This lneans, however, that L3 B ° is 
lnapped to (LBO cl(a«)) B via ç whore ].ri : 3, x(0) : 70, x(1) : 7) and 
x(2) : sllp{.s(0) + 1; x(0) < Ç}. Next assume a" is relevant and considor 
0 «r(ll-). 
Assulne &B ° ¢ . Then there is s  T such that [s = Ix I- 1 and x s = 
" r ° and ,le  B ° is mapped to ,& B  via ç because 
In case s  S, we have ms =  s  
0 l,a v(ll ) < (11- ). 
z e n for l,y  e g0 with I1 = *, r(ll- ) = « - - 
[11 Ce 8  T  S, let j0 ( 18] lle lnaxi,nal with sU0 e S. Defilm g by 
g[([ - 1) : d s "n and g(]l - 1) : x(Ix ] - 1) and note that .1«.  B ° gets mapped to 
u  B  provided wc can show  is relewmt. In case jo > 0 there is nothing to show 
lmcause whelmver 0   =*(3) > w. and. 
a'(3) > wt. J  j0 even, tht'n also .r s(j + 2) 0 - 
. " " (j0) > w while, if J0 is odd. 
if j0 is even. we additiolmllv have a s (30) = SUl)a xs 
we additionally lmve s(3o + 1) = 7sjo > w. In case j0 = 0 this is true because 
.,'(1)  Sï  and 00 is larger than ail the 0 for which a'g(j)  S where j > 1 is oaa. 
Even though B  is not a ri'ce, xve can verify, as in the proof of Lelnlna 3.1. that. 
letting Ç := cl(B), Ç  Lmad = ,eT Comp, and that ç = ' : Ça + C, 
et < w, which extends ç and lnaps Comp« to Comp, is a weak isonorphisln. 
B)- Lelnlna 3.2, ?[(' and F[C are isomorphic bv a map  = '. X sends 
[Ca-lmlnes to [C-nalneS, and we defilm .n to be the image of .a under 
Bx- constructioll, it is then also ilnllmdiate that whenever  < , we can find 
o < Wl such that B  U B  and B a U 3 are weakly isolnorphic via the lnapping 
fixing nodes of B  and sending the x to the corlespondilg x, and such that 
this lnapping identifies cofinal subsets of the traces of Z on the two sets. m Again. 
this weak isomorphism canolficallv extends to a weak isomorphisln of C U Ç and 
C a U C, which in turn means that F[C U C  and F[Ca U C  are isolnorphic 
(Lemlna 3.2) bv a mapping sending the llalne .4 to a. Since À  and . are 
forced to be ahnost disjoint (by F[Ca U C), so are  and . ' (by the iSOlnorphic 
[C  U Cç). Since [C  U C embeds into [L (Lelnlna 1.1), this completes the 
proof of the non nmxinmlity of  and, by Çorollary 2.2. of the Main Theoreln. 
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CYCLICITY OF CM ELLIPTIC CURVES MODULO p 

ALINA CARMEN CO.IOCAIU 

ABSTRACT. Let E be an elliptic curve defined over Q and with complex multi- 
plication. For a prime p of good reduction, let. Ê be the reduction of E modulo 
p. We find the density of the primes p <_ oe for which E(Fp) is a cyclic group. 
An asymptotic formula for these primes had been obtained conditionally by 
J.-P. Serre in 1976, and unconditionally by Rare Murty in 1979. The aim 
of this paper is to give a new simpler unconditional proof of this asymptotic 
formula and also to provide explicit error terres in the formula. 

1. INTRODUCTION 
Let E be an elliptic curve defined over Q and of conductor N. By a famous result 
of Mordell, the set E(Q) of Q-rational points of E is a finitely generated abelian 
group. The study of the free part of E(Q) is still one of the major problems in 
arithmetic geometry. 
Now, for a prime p of good reduction for E (that is, p  N), we denote bv E 
the reduction of E modulo p. This is an clliptic curve defined over Fp, the finite 
field with p elements. Naturally, as in the rational case, one is interested in the 
study of the structure of the group E(p) of Fp-rational points of E. From classical 
theory, E(Fp) can be written as the product of two cyclic finite groups. Indeed, 
E(Fp) C_ E(Fp)[k] C_ Z/kZ Z/kZ, where Fp denotes the algebraic closure of Fp, k 
is a positive integer such that the order #E(Fp) of E(Fp) divides k, and E(Fp)[k] 
denotes the group of Fp-rational points of E annihilated by k. Early computations 
of Borosh, Moreno and Porta ([BMP]) showed that, in fact. for "many" primes p. 
the group E(]p) is cyclic. One expects this to be true for infinitely many primes 
p, as suggested by the elliptic curve analogue of Artin's primitive foot conjecture 
formulated by Lang and Trotter in 1977 (see [LT2]). 
Out goal in this paper is to provide an asymptotic formula, with explicit error 
terres, for the function 
f(x,Q) := #{p <_ x " p ,,E(Fp) cyclic}. 
in the case of an elliptic curve E defined over ( and with complex multiplication. 
In 1976 (see [Sel]), J. -P. Serre showed that C. Hooley's conditional method 
of proving Artin's conjecture on primitive roots (see [Ho, ch. 3]) can be adapted 
to estimate f(x, (). More precisely, let ( denote the algebraic closure of ( and 
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let Q(E[k]) denote the fiel(l obtain(,d by adjoining to Q the coordinates of the 
Q--rational points of E7 annihilated by k. T]lel, under the Generalized ienann 
Hypothesis (denotcd GRH) tbr the Dedekind zeta ftmctions of the division fields 
Q(E[k]) of E, Serre proves that, as ,r + OE, 
(1) f(x,Q) = fli,c + o . 
where li,r := f" 1 dt is the logarithmi« integral and 
k--I 
with p(-) denoting the M6bius fmlclion. 
X> recall hal for real-valued fimctions f and 9 # 0 we write f(z) = o(9(«)) to 
lllean that lim. I(*} _ O. Also. if 9 has p,,sitive values, we write 
g(x) -- 
()r f « g fo mean that there exists a 1)ositive constant 
Ag(x) Vx. If lhe COllSlallt ,4 d('i)ends on S()llle qnantity B. then we may write 
/(x) = ()(9(x)) or f « g. In this paper, wllenever we write f(x) = O(g(x)) or 
.f « 9, we nlean thal thc imi)lied O-constants are absolute. If f « g « f, then we 
write f  9. 
In 1979  (sec [Mul. pp. 161-167]), Rare Murtv removed GRH in formula (1) 
tbr elliptic curves with complex multil)lication (deuoted CM). His proof uses clss 
fiekl theoretical properties of the division fields of CM elliptic curves, as well as a 
number fiekl version of the Bombieri-Vinogradov theorem (whose proof is based on 
the large sieve tbr number fields), lu 20(10 (sec [acC1]), the author proved formula 
(1) for elliptic curves without complex multiplication (denoted non-CM) under the 
assumption of a quasi-GRH (more precisely, a zero-free region of real part > 3/4 
for the Dedekind zeta functions of Q(E[:])). For more history about /(x,Q) in 
i)oth the CM and non-CM cases we refer the reader to [acC1], [aeC2] and [Mu3]. 
In this paper we give a llPW simpler unconditional proof for the asymptotic 
formula for f(x, Q) in the complex multiplication case, and provide explicit error 
terlns in this fornmla. ç are proving the following: 
Theorem 1.1. Let E be a CM elliplic curve d«fin«d over Q, of condctor N ad 
with complex multiplicatio by the full tin9 of integers 0- of an imaginaw quadratic 
fi'eld Iç = Q ( ) . wh«re D is a positive square-free znt«ger. Then, as x 
(2) f(x,Q)=IElix+O.v (logx)(logloglogx) " 
or, more precisely, 
(3) f(x,Q)=IElix+O (log x)(log log og,  1Og log x . 
where the O-constant in (2) depends on N and the one in (3) is absolute. 
Corollary 1.2. Let E be a CM elliptic curve defined over Q, of conductor N and 
sch that Q(E[2]) ¢ Q. Then the smallest pme p { N for which Ê(Fp) is cyclic 
. ,« o (xp () ) . r v«e O-co.  

lit, was communicated to the author by Rare Murty that this result was obtained in 1979; 
however, it appeared in print onl3r in 1983. 
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It is possible that the error t.erlllS iii Theorenl 1.1 Call be ilnproved, but this 
involves lnore sophisticated lnethods than the ones used iii our pal)er. \Ve relegate 
this te ftltllle research. 

2. PRELIMINARIES 
2.1. Notation. Given ai1 elliptic curve Æ defined over Q, p will always denote a 
prime of good reduction for ?. \Ve set ap := p + 1 - E(IFp) and say that p is of 
ordilmrv reduction if ap  (, ami of supersilgular reduction if ap = 0. We denote 
bv zcp and  the roots of t|le po|ynomial ,\' - aw\" + p  Z[X]. 
If hot otherwise stated, q will denote a rational prime and k a positive integer: 
n(x) wi|l denote t|le nmnber of ratiollal prilnes G ,r; S will denote the cardinality 
of a set S: Ker ç wil| denote the kernel of a morphism ç. 
2.2. Algebraie prelinfinaries, The followmg prelilninary |enmlas are well kmwn. 
but, for the sake of completeness, we inçlude thenl hel'e. 
Lemma 2.1. Let E be an elliptic curve defined over Q and of condu«'tor N. Let 
E[k] be the gro'u 1, of k-divi.sion points of E. Then 
(1) the ramified p,'in, es oJ'Q(E[k])/Q are divisors of hN; 
(2) assuming that E bas co'mplex multiplication and !« > "2. we hat,e 
O(k) " << [Q(E[k])-QI << k 2, 
where çb(k) dezotes the Euler flm«tiou. 
For proofs of this lelnlna the reader is referred te [Silvl p. 17"9] and [Sllv_. 
p. 1351. 
Lemma 2.2. Let E be an elliptic curve defined over Q and of conductor N. Usin9 
the notation introduced in Section 2.1 we bave that, for a positive izde.qer !; and a 
prime p !; of 9ood redu«tion for E, 1» splits completely in Q(E[k])/Q if and only if 
zrp-- 1 
 is an algebraic integer. 
Pro@ \Ve recall that 7rp is the algebraic quadratic integer corresponding te the 
Frobenills endonlorphislll 
which we also denote by 
Since (p, kN) = 1, we have that 1' is ulralnified in Q(E[k])/Q (see part 1 of 
Lemna 2.1). By classical results in algebraic nulnber theory, p splits completely in 
Q(E[k])/Q if and only if 7rp]oe[] = 1, where 1 denotes t.he identity map. This last 
condition is equivalent te saying that Ker([h]) C Içer(wp- 1) as nmps Ê (-7)  
Ê (-7), where [k] is the multiplicatiol, by k map. Hence there exists an elliptic 
clIrve endomorphism b" E (IFp)  Ê (-7) such that b o [k] = rrp - 1 (see [Silvl. 
Crolla.ry 4.11. p. 77]). This is equivalelt to saying that '- is ail a.lgebraic 
k 
integer. [] 
Lemma 2.3. Let E be a CM elliptic curve defined over  an.d with complex mul- 
tiplication by an imagincwy quadratic field Iç. Then. for every prince p of ordinary 
good reduction for E, we have Q(Trp) = K. 
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Pro@ First we observe that 
Q(rp) c_ Endp(E) ®z Q c_ Endw(E ) ®z Q. 
Then we note that, since E has complex multiplication by/x, we have an embedding 
K C Endw(Ê ) ®z Q, and, moreover, since p is a prime of ordinary reduction, we 
actuallv have K = Endw(E ) ®z Q. Thus Q(rrp) c K for any prime p of ordinary 
reduction for E. But K is a degree 2 extension of Q, and so is Q(rp). This gives us 
the desired equality. [] 

Lenlma 2.3 describes a feature of Chi elliptic curves that will play a very impor- 
tant role in our unconditional estimates of f(x, Q). It actually describes one of the 
main differences between Chi and non-CM elliptic curves (see [LT1]). 

2.3. Analytic prelilninaries. The next prelininary lemma is an application of 
the sieve of Eratosthenes, which we recall below. 

Theorem 2.4 (The sieve of Eratosthenes). Let .A be a set of natural nrnbers <_ x. 
and let 7 ) be a set of rational primes. To each pmme p 
distinguished residue classes modulo p. For any square-free integer d composed of 
primes of 7) we set 

.A(d) := {a  .A  a belogs to at least one of the w(p) residue classes 
modulo p for all 

and 

() := 1-[ (P)" 
p[d 
For a fixed real number z, we let S(.A. 7), z) be the number of elements a  .A that do 
not belong to any of the distinguished residue classes modulo p for all p  7), p <_ z. 
and we set 

We assume that 

(1) there exists a real number X such that, for all square-free integers d com- 
posed of primes of 

(2) 

#A(d) = X- + Rd 

for some Rd = O(w(d)); 
 w(p)logp _< clog z + O(1) for some positive constant c 
P 
pP 
p<_z 

Then 
S(.A,7),z) = XI"(z) + O (x(log z)C+l exp (--- 
where the implied O-constant is absolute. 

log 

For a proof of this result, see [Mu4, p. 141]. 
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Lemma 2.5. Let x  ]R and let D. k be fixed positive integers with k < v @ - 1. 
Then 

Sî := # p<_:r'p= k+l +D l'2forsomect,/3Z 
: O((-.X+l) v/71°gl°gx ) 
kv log --: --A " 
The implied O-constants are absolute. 
Proof. 1. Let us observe that the conditions p  x and p = (ok + 1) 2 + D/32k 2 for 
some a, fl G Z imply 

Thus 
() s _< }2'# { 
where the sure '¢ 

-1 - x/Tl`. , -1 + v/7]l`: 
V V 
»v' 'v 

çZ, 

nz, #0. 

 [ -1-v'@k '--l+v@]k rZ'(ah+l)2+D/3212apri'ne}" 

a E k ' k flZ , 

7) := {pa rational prime" (p,k) = l, (pD) = l}, 
with (}) denoting the Legendre sylnbol nlodulo p. To each prinle p  7) 
we 
asso- 
ciate the residue classes 
(-1 + [3kD)k -1 (modp), 
where D is an integer such that D 2 = -D(modp) (let us observe that (ck + 1) 2 + 
D132q 2 = p imposes the conditions (=) = 1 and (p,k) = 1. and hence D and 
k-(modp) are well defined). 
For a fixed real number z we have 
 oe . ,  V1Z'(at,'+l +D/32k 2aprime 
(5) < s(A,V,z) + ,(z) 
< S(A, 7), z) + z, 
with S(A, 7 ), z) defined as in the sieve of Eratosthenes. 
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Now we want to verifv that the hyl)otheses of Theorem 2.4 are sa.tisfied. Ele- 
mentarv estimates givc us 
#.A(d):=#{ce.A'(ok+l)2+D1321«2=O(nmde)}=2 +1 
for ail square-free integers d composed of primes of P. Thus the first hypothesis 
of the sieve of Era.tosthenes is satisfied with w(d) = 9 and X =  
-  + 1. Using 
glertens" theorem and recalling tha.t () = 1. hence that p splits completely in 
 ,(v) ov _   og, _g: + o(1). 
vz pz 
Tlms hc sccoud hyl»ohcsis of lhe sicvc of Eroshcnes is stisfied wih c = 1. 
Thercfore, 
+l ll'(z)+O  +l(logz) 2exp 
k log z " 

Il {z) = 1 - _< exp -2 « exp(-loglogz)- loz' 

by using the elemeutarv inequality 1 + t <_ exp(t) and. a.gain, Mertens theorem. 
Let us choose z such that 

Then 
O(X/-t- 1 
k 
and so 

s(A. p. z) 

log z -- 

3 log log .r" 

(log log x  
: (--. -t- 1) O \,og  j 
-- (+,)o. 

V+I 1 ) 
k log.r(lo log x) 2 " 

(  ) 
+ 0 k log x(log log x) 2 

From (5) we obtain 

= (. + 1) 0 ( l°-g l°g'r log __2!_._ ) , 

which, used in (4), completes the proof of the first part of the lemma. 
2. Sinfilar to the proof above 
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We relnm'k t.hat for SI. and Sî. of the al)ove lelnlna we aetuallv have elelnentary 
log log  
estimates that are weaker than t.he ones given bv Lelnlna 2.5 onlv by a 
log x 
factor. The sieve has been invoked pre«isely fol" obtailfing this saving. 

Lemma 2.6. Keeping the notation of Lemma 2.5, we bave that, for any k and x, 

, () 
t. «< /,.-- +1 , 

where 1 <i<2. 

Pro@ We justify this estilnate for i = 1. The case i = 2 is l'esolved similarly. We 
observe hat the Colditions p <_ .r and p = (ak + 1) 2 + D/32k 2 for some a,/3 G Z 
give us - + 1 choices for o raid  choices for/3. The lemlna follows. [] 

3. THE PROOF OF THE THEOREM AND COROI,LARY 

As explained in [Mul. I,I»- 153-154], we have that E(IF») is cyclic if and only if 
p does not split COlnpletely in Q(E[q]) for any prime q ¢ p. Also, we have that if 
p _< .r and p splits eompletely in Q(E[k]) for SOlne /,', t|len k21(p + 1 - ap), and so, 
using Hasse's bolnld ap <_ 2V@, we obtain k _< 2v. Therefore, using the simple 
asylnptotic sieve, we tan write 

fez, Q) = N(x,g) + 0 (lll (x,y, 2v#-) ) , 

where 

]V(z,y) := 4¢ {P _< x" pdoes llOt split COlnpletely iii ally Q(E[q])/Q,q <_ y}, 

11I (x,v. 2v') := # {p < x" psplits conlpletely in someQ(E[q])/Q 
with .q < q < 2V}. 
and where y is a real nulnber to be chosen later, hl order to estinlate f(x, Q) we 
need to estimate ea«h of N(x, y) and ]Il(x, y. 2v ) and to «hoose the paralneter y 
appropriately. 

3.1. Estimate for N(x, y). By the inchlsion-exchlsion principle we have 
]V('F,V) = Et/-/(/')71-1( x" Q([/çl)/Q), 
where the sure is over ail square-free positive integers k < 2v whose prime divisors 
are < y, and where 
rl(z, Q( E[k]) /Q) := #{p < x "p splits COlllpletely in Q(E[k])/Q}. 
We estimate this sum by using the unconditional effective version of the Chebotarev 
density theorem as stated in [Mu2, p. 243] or [acC1. p. 337]. To do so, let us recall 
froln [Se2, p. 130] that if L/Q is a finite normal field extension that is ramified only 
at the primes p, p2,..., Pr.. then 
IL" Q-- l°g I disc(L/Q)l -< log[L" Q] + E logpj, 
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where [L'] and disc(L/) denote the degree and the discrilninal,t, respectively, 
of L/. V'e apply this result, together with Lemma 2.1, to the fields (E?[k]), whose 
degree and discrinfinant over  we denote by ri(k) and d, respectively. D,'e get 
n(k)ldkl ' EE k8N 2 

and 

,,(k) 00g 141) 2 « k6 00g (»2v))2, 
and so the maximum of the two quantities above is « k8N 2. In order to apply the 
mmonditional effective Chebotarev density theoreln mentioned before we need to 
have kN  « log x. Since k  exp(2y), it is enough to choose 
(6) y = (log log x - 2 log N). 
Then, by the unconditional effective Chcbotarev density theorem, we obtain 
N(x, y) = n(k)] lix+ O a'exp -A 
tbr some effective positive constant A. To handle the error terre we use that n(k) « 
k 2 and that there m'e at lnost 2 .v square-free numbers composed of primes  y. Then 
(r) N(x,y)= ç n(k)]lix+O N/4(logx) B 
for any positive constant B. 
3.2. Estimate for M (x, y. 2). For rem numbers {, {2, we denote by 
ao («.{.{) 
the number of primes p  x such that p bas ordilmry reduction and splits completelv 
in some Q(E[q]) with {  q  {2, and by 
M(x.{.{) 
the lmmber of primes p  x such that p has supersingular reduction and splits 
completelv in some Q(E[q]) with {  q  {2- X write 
(s) M (, , ) = a o (x.v. ) + a * (,. v. e) 
and estimate each of the two terres. For the first one we observe that 

(9) 

where 

y<q_<2q 

rcï(x,Q(E[q])/Q) := #{p <_ x" ap J: 0 and p splits completely in Q(E[q])/Q}. 

By Lenmms 2.2 and 2.3 we obtain 
,ï(x,Q(F[q])/Q) < # {p < 
Since the norm of rp in K/Q is p, we get 
{ p -- 1 
# px. 
q 

X ° -- 

rcp - 1 " 
 (DI,- . 
q 
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where Sq is Sq  if-D = 2,3(mod4), and ,çq if-D = l(mod4), with Sq, Sq 2 as in 
Lelnma 2.5. 
Let us fix a real nulnber "u < vif- 1. Using the elementary estinmte for Sq given 
in Lelnnm 2.6, we obtain 

Z rï(.r, Q(E[q])/Q) <_ 

(10) << 

 Sq 
  (7 1) 
<q<2/ qv + 

2x 1 v 
u<q<_2x/' 
x  log log a" 
+ 
v"u log u 

1 

On the other hand, using the estilnates for Sq given in Lemlna 2.5, we obtain 

y<q<_u 

Sq 
y<_q<_u 

y<qSu 

We choose 

u = log x 
*(loglogx 91ogN) (see (6)) and that D is bounded, since E 
and recall that y = g - _ 
bas Chi. Then, from (9), (10) and (11) we get 
(12) M°(x,g, 2vfT) =0 og   
(logx)(log N= (loglog v= ) 

For the second terln in (8) we have 

(3) 

where 

r(x,Q(E[q])/Q) := #{p _< x: ap = 0 and p splits completely in Q( E[q]) /Q }. 
We observe that if p is a prime of supersingular reduction that splits completely in 
some Q(E[q])/Q, then q = 2. Indeed. for such primes p and q we have, on the one 
hand, that q](p + 1 - ap) = (p + 1), and, on the other hand, that q](p - 1): thus 
q[2. Now we note that in the sure of (13) we run over q > y; thus, by out choice of 
y (see (6)), q  2. This implies that 
(14) M'(x,y,2v) =0. 
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#(k) | li X. 

--, t,(/) t,(k) V.,, .(k) 
» ,,(») -  ,,(.) /--- ,,(»)' 

where k" 
which there exists a prime diviser q > y. Using part '2 of Lelmna 2.1 we get that 

lneans that the Slllll is over those positive square-free integers k for 

"p(k) x 1 
lix «< log----TZZqt3/2 
 q>y t=l 

log :r log :r " 
-- ) 0og log -- ) 

.1" 

\ 
log log x ] 
log x J " 
log W- 

T}IllS 

(log x)y log B 
= O ((logx)(log 

=  log x 
(15) f(:r,Q) Ielix +O (l°gz(l°gl°gw-) 
This completes t he proof of Theorem 1.1. 

3.4. The proof of Corollary 1.2. First, let us recall that it was pointed out bv 
Serre (see [Mu3, p. 327]) that the density fE is positive if and only if Q(E[2]) ¢- Q. 
Now, we note that a necessary condition for forinula (15) to hold is that x _> 
e.p (V). Shen, if  × e.p (V). the ,,in terre of (15) ,,-iii be bigger than the 
error terln. This proves the assertion of the corollary. 

4. CONCLUDING REMARKS 

As mentioned in the proof of Corollary 1.2, the density [E is positive if and only 
if Q(E[2]) ¢- Q. For the sake of clarity, we explain this in what follows in the case 
of a CM elliptic curve/3 defined over Q. Naturally, in order te have Ioe ¢- 0 we need 
te assume Q(E[2]) ¢ Q, for otherwise the torsion part of E(Q) contains the Klein 
four group and se E(Fv) calmer be cvclic. The condition is also suflàcient. Te see 
this, let us first note that if Q(E[2]) ¢ Q, then [Q(E[2]) -Q] is 2, 3 or 6. We let 
K be the Ulfique abelian subextension contained in Q(E[2]). Also, we let Iç be the 
CM field of E. We recall that Iç(E[q]) = Q(E[q]) for any prime q > 3 (see [Mul, 
p. 165, Lelnlna 6]), and we observe that since K is a quadratic field and Iç2 is a 
cubic or a quadratic field, we have either K ç? K = Q or Iç = K. If K VI K = Q, 
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then using that K2 ç Q(E[2]) and Ix" ç Q(E[q]) for anv q _> 3. we deduce that the 
densitv of the primes p that do hOt split completely in anv of the fields Q(E[q]) is 
greater than or equal to the density of the prilnes p that do not split COlnpletely in 
K 2 and N. In other vords, 
( )( )» 
c   [ç.] 1 [¢.] _ . 
If N2 = N, then K ç Q(E[q]) for any prime q. and so the ,h'nsity of the primes p 
tha¢ do hot si,lit complotely in any of the fiehls Q(E[q]) is greater than or equal to 
the density of thc primes p that do hot si,lit COlnI)letely in N. In other xvords, 
( I ) , 
f > 1 [Iç Q >- 
This completes the proof of the positivity of . 
The main significance of our uncomtitional proof of the asymptotic formula for 
f(x,Q) in the case of a ('gl elliptic curve lies in thc silnpli«ity of the tools that 
are used. Rare Mm'ty's initial proof avoided the (IH bv using a di«ult appli- 
cation of the large sieve for lmmber fields, nalnely a Bombieri-Vinogradov type 
result for mmfl)er fields. In our nev proof we use instead an application of the 
sieve of Eratosthenes, one of the silnt)lest sieves in nmnber theory.  point out 
that this application of the sieve of Eratosthenes (Lemma 2.5) could be viewed as 
a Brun-Titchmarsh theorem for quadratic mmfl)er fields, since it gives nontrivial 
upper bomds for the lmllfler of (prilwipal) prilne ideals whose gelmrator satisfies 
congruente conditions. A result of this kind had been obtained in [Sch], but as an 
application of the large sieve for mmber fields, and «ould hm'e been used in our 
treatment of M(z. g. 2). 
Another significance of our new proof is that it provides explicit error terres, 
with absolute O-constants. As noted in Corollary 1.2. we can then deduce an 
unconditional upper bound for the smallest prime p for which E(Fp) is cyclic. 
Considerable improvements of this bound, mder GRH. will be discussed in an 
upconfing paper. 
Naturally, one can ask if our ideas can be explored finther and used in other 
related situations. For example, one could consider the question of determining 
the number of prime ideals for which the reduction of a CM elliptic curve defined 
over a number field gives a cvclic group. It seems that out tools can be used iii 
this situation. Another question is that of using the ideas of this paper in the case 
of a non-CM elliptic curve. At present no unconditiolml proof for the asymptotic 
formula for f(z, Q) is known in this situation, but, as mentioned in Section 1. onlv 
a proof based on a quasi-GRH assumption (see [acC1]). If we assume a variation of 
a conjecture of Lang and 'otter on the number of distinct fields Q(gp) obtained 
when p runs over primes of ordinm'v reduction for a non-CM elliptic curve (see 
[LT1]). then it turns out that we can follow the current CM approach even in 
1 
the non-C case. The dependence  on the discriminant D of the estimates 
provided by Lemma 2.5 vill be more advantageous than the dependence on D 
provided bv Schaal's result memioned above. This is. again, ail asset of our new 
proof. Yet another related question is that of determilfing an asymptotic fornmla 
for the number of primes p for which the order of E(p) is square-free. The ideas 
of our paper can be successflflly used to answer this question if E is a Cg[ elliptic 
curve. The details of our last two claires will be given in different upcoming papers. 
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TAYLOR EXPANSION OF AN EISENSTEIN SERIES 

TONGHAI YANG 

ABSTRACT. In this paper, we give an explicit formula for the first two terres 
of the Taylor expansion of a classical Eisenstein series of weight 2k ÷ 1 for 
F0(q). Both the first ternl and the secoud terre have interesting arithmetic 
interpretations. \Ve apply the result to compute the central derivative of some 
Hecke L-functions. 

0. INTRODUCTION 
Consider the classical Eisenstein series 
 Iln(7"r) s, 
which bas a simple pole at s = 1. The wcll-known Kronecker limit formula gives 
a closed formula for the next terre (the constant terln) in terlns of the Dcdckind 
rkfunction and has a lot of applications in nulnber theorv. It seems natural and 
worthwhile to studv the saine question for more general Eisenstein series. For 
example, consider the Eiscnstein series 
(0.1) E(r,s) :  e(d)(«r + d) -2k-' I,n(Tr) -k. 
-eP\Po(q) 
Here 7 = ( d b ), --q is a fundamental discriminant of an imaginary quadratic fiel& 
and e = (-q). This Eisenstein series was used in the celebrated work of Gross and 
Zagier ([GZ, Chapter IV]) to compute the central derivative of cuspidal modular 
forms of weight 2k + 2. The Eisenstein series is holomorphic (as a function of s) at 
the symlnetric center s = 0 with the leading terre (constant terre) given by a theta 
series via the Siegel-Weil formula. The almlogne of the Kronecker limit formula 
would be a closed formula for the central derivative at s = 0 -the main object 
of this paper. This would give a direct proof of [GZ, Proposition 4.5]. Another 
application is to give a closed formula for the central derivative of a family of Hccke 
L-series associated to Chi abelian varieties, which is very important in the arith- 
metic of Chi abelian varieties in view of the Birch and Swinnerton-Dver conjecture. 
This application will be given in section 4. We will also prove a transformation 
equation for the tangent line of the Eisenstein series at the center, which should be 
of independent interest. 
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To make the exposition silnple, we assume that q > 3 is a prime congruent to 3 
lnodulo 4. Let k = Q( 
Set 
A(s,) = -V( s+ 1)L(s, ) 

(0.2) 
and 
(0.3) 

/*(T,) 
= q  A(s + 1, «)E(T, 
It is well known that E*(r, s) is hololnorphic. 
As in [GZ, Propositions 4.4 and 3.3], we define 

k 
(0.a) p(t) =  ()  (-t),,, 

alld 
(0.5) qt.(t) __ e-t(u- 1)ku--du, t > 0. 
\;e rena.rk that p(-t) and qt.(t) are two «basic" solutions of the differential equa- 
tions 
(0.6) ,C"() + ( + t)C'() - »C() = 0. 
Filmlly. let p(n) be given by 
(0.7) <() =  (,,)- 
Theorem 0.1. Let the notatwn be as above, and let h be the ideal class number of 
k. lI'rite r = u + iv. Then 
E*(r, 0)= v-(h + 2 __,p(n)p(4rcnv)e(nr)) 

and 

and 

ao(v) = h 

( A'(1, e)  1) 
log(qv) + 2 A(1, e + Z  

a = (ordqt + 1)p(r01og q +  (ordpr + 1)p(n/p)logp. 
(E)=--I 
The forlnulas should be colnpared to those for  in [GZ, Propositions 4.4 and 
4.5]. In fact, lnultiplying our forlnulas by the theta function in their paper and 
taking the trace would yield their formulas for ç. The method used here seelns 
to be more suitable fol" generalization. The proof is based on the observation that 
the Eisenstein series (0.1) can be split into two Eisenstein series. One of them is 
coherent, and it is easy to compute its value. It contributes little to the central 
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derivative. The other one is incohel'ent, contril)utes nothing to the value, a.nd its 
central derivative can be computed by the method of [KRY], where we dealt with 
the case k = 0. This consists of sections 1 and 2. 
In section 3, we study how the value and derivative behave mder the Fricke 
involution "r  -1/qr and obtain the following fllnctional equation. One inter- 
esting point about the equatiou is that it basically follows from the definition of 
antomorphiç fornls (sec (3.2)). 
Theorem 0.2. The mo&dar forms E*(r, O) and E*'(r, O) satis.[y tbe.[ollowmg flmc- 

tioal eq-uatio: 
* 1 
( (-.o) 
E*'( -1 ol 

= i(v) 2"+ 
j=l 

--1 
_1+1 log q ,c?*'(T, 0) " 
3 

Finally, let p be a canonical Hecke charact«r of weight l of k (see section 4 for the 
definition). It is associated to the C elliptic çurve 4(q) studicd bv Gl'OSS ([Gro]). 
When q = 3 mod 8, S. Miller and the author proved re«entlv lhat the central 
derivative L'(l. 13) 7 0 ([MY]). Since thc central derivative encodes vcry important 
information in the arithmctic of A(q), it is iml»ortalt to film a good formula for the 
central derivative. Standm'd calculation shows that the L-series L(s. I 3) is E(r, 2s) 
evaluated at a ÇM cycle. So Theorem Il.1 gives an explicit forlmlla for the central 
derivative L(1,13) (Crollary 4.2). 

1. COHERENT AND INCOHERENT EISENSTEIN SEl:lIES 

Let G = SL2 over Q, and let B = TN be the standard Borel subgroup, where 
T is the standard maxinml split torus of B and N is the unipotent radical of B. 
Their ratiolml points are given bv 

and 

a-  a } 

Consider the global induced representation 
I(s. e) 
= IndB(A) ] ] 
of ((N), where N is the ring of adèles of Q. Bv definition a section (s)  I(s. «) 
satisfies 
(1.1) (n(b)m(a)g. s) = «(a)lal+(g, s) 
for a  A* and b  A. Let K = SL() and let K = SO(2)(). Associated to a 
standard section . which lneans that its restriction on IçIç is ildependent of s. 
one defines the Eisenstein series 
(1.2) E(g,s,) =  (2g, s). 
eB(Q)ça(Q) 
It is absolutely convergent for e s > 1 and has a lneromorphic continuation to 
the whole complex s-plane. X consider three standard sections 0, i in this 
paper. For every priine p { q, let p  I(s. ep) be the unique spherical section 
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such that (I)p(x) = 1 for every ,r C Kp = SL2(Zp). Let (I) C I(s, e) be the unique 
section of weight 2k + 1 in the sense that 
(1.3) (gko, s) = .(9, s) e(+)0 
for everv ko ( cosO sinO 
 = -iOcoO) K" Forp=q, let 
4= cq 
be the Iwahori subgroup of Kq. Then % defines a character of .lq via 
(1.) «( cq 
As described in [KY, section 2], the subspace of I(s. Q) consisting of % eigenvec- 
i determined 
tors of Jq is two-dimensional and is spammd by the cell fimctions of q, 
by 
(1.5) 
 qw/, s) = 5 0, where w0 = 1 and u, = w = . 
ç denote this subspace bv II (,lq, eq, s). A better basis for this subspace turns out 
to be given 
1 
(1.6) çq 
which are "eigenflmctions" of somc intcrtwining operator (see Lemma 2.2). Set 
(1.7) 0 o 
= q   and = q  p. 
Clearlv, 0 1 + -). For +iv with v 
. = ( + r=u >0.1et 
(1.8) g = ,,(),(). 
Then standard computation gives 
Proposition 1.1. Let te notatio be as above. Then 
.(, )= .-},(g, , 0) 
= -,,--i(*(g,,+) + *(g ,-)). 
2 ' 

Here 

E*(g,s, ep) = q A(s + 1,e)E(g,s, dp) 
is the completion of the Eisenstein series E(g, s, (I)). 
As we will see in Proposition 2.4, the Eisensteiu series with (I) + behave almost as 
"even/odd" functions respectively, and both have nice functional equations. This 
is hot a coincidence. Indeed, ri'oto the point of view of representation theory, 
 +(g, 0) is a coherent section in I(0, e) in the sense that it cornes from a global 
(two-dimensional) quadrat.ic space, while (I)-(g, 0) is an incoherent section in I(0, e), 
coming from a collection of inconsistent local quadrat.ic spaces. We refer to [Ku] 
for explanation of this terminology and for a general idea for computing the cen- 
tral derivative of incoherent Eisenstein series. Every section in I(0, e) is a linear 
combination of coherent and incoherent sections; we just made it explicit in this 
case. 
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2. PROOF OF THEOREM 0.1 
Let 0 = I-I 0p be the "canonicar' additive character of A via 
{e 2rix if p = oe, 
(,p(x) = e_2( ) if p -¢ x. 
Here A is the canonical map Qp  Qp/Zp "- Q/Z. For a standard section 
= I-I (p E I(s, e) and d E Q, one defincs thc local Whittaker flmction 
IVa,p(9, s, ¢) =  ¢(u,.,(b)9,s)Op(-db)ab. 

(2.1) 
Let 
(2.2) 

be its completion. \Ve also set lp(s) 
M M (s) is a nornlalized intertwning opcrat or from I(s. ) to I (-s, ). 
In gencral, an Eisenstein stries E*(9, s, ç) has a Fourier ext)ansion 
(2.3) E*(g,s, 
d 

with 
(2.4) 

E(9, s,4P) = q 
P 
for d ¢ 0 and 
 -,-a «,  
(2.5) E(9, s,ç)=q  A(s+l,¢)ç(g,s)+q  5I (s)ç(g,s). 
The local Whittaker integrals are computed in the next three lcmmas. 
Lemma 2.1 ([KRY, Lemma 2.4]). For a flnite prime number p ¢ q. 
II a,p1, s, p) = 0 vless ordp d > 0. h such a case, one bas 
ord. d 
v%,(1. . %) =  (,(v)p-)  
r=O 

and 

M(s)((s) = Lp(s, ()(p(-s). 
Here çp is the unique spherical section defined in section 1. In particular 
where pp(d) -=- p(pord»d) for p < oe. 
Lemma 2.2. For p = q. one bas 

lV*,(Wl, s. '+ = 

one as 

(l+(q(d)q-S(°rdqd+l)) (±--- ) 
\ --q 
0 

if ordq d _> 0, 
if ordq d = - 1, 
otherwise 
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Mq(s)q 1 + 
= -'--------q . 

Pro@ The tir,st tbrnmla follows flore [KIY, (3.26)-(3.29)]. For the second formula, 
notice that AI,(s) is an itertwinig operator between eigenspaces ll'(Jq, eq, s) and 
:(./. ,-) of .,. 
.   + b 
.I, (S)q = a q + 
for sonc constants a  and b . Phgging in 9 = u'o and t,, and appl)ing the first 
formula, one gets the desired fbrnmla. 
Lema 2.3. Let  =  be the local section in I(s. e) defined b9(1.3). 

Il(g. l. t3, ) = / e-gx(._  31- h) a-1 (.F - h)O-d.Æ 
-Fh>0 
is Shi,tura  «la fu,cliot for 9 > O. h  N, atd e  and e  s'ucietttl9 large 
(2) For d > O. o« bas 
1 
ll.(g.0. ) = 2it,pk(4dt,)«(dr), 
where pk is deflned b (0.4). 
(3) For d < O , one has ll..(g.O, ) = O. and 
1 
1I 3. (g. o.  ) = i,, q»(-4dt,)e( dT). 
where q is 9i,,e, by (0.5). 
 j-sC 
(4) oE()() = i =0 +/2(-)e(-)  
Pro@ The proof is the sae as that of [KRY. Proposition 2.6] and is left to the 
reader.  

Proposition 2.4. 

(2.6) 

k k 
[[(j - )F*(g,-, e ) = : H(J + )F*(r,., 

Pro@ By Lenmlas 2.1-2.3. one has 

Now the proposition follows from the fimctional eqta.t.ions 
and 
E(9, s, ) = E(g,-s,l(s)). 
Here I(s) = M*(')A(s + 1. )- is the unnormalized intertwining operator from 
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Theorem 2.5. One bas 
v (T,o., +) = 2(hq+ 2 

('.8) 
atd 
(2.9) 

1 *l -- 

Proof. First. we observe that 
(2.10) H pp(d)(1 + %(d)) 
SillCe 

= p(Idl)(1 + %(d)) = 2p(d) 

1 = H e(d)= sigl,(d)eq(d)II(-1) 

poe pi d 

Fornmla (2.8) is a special case of the Siegel-Weil forlnula. \Ve give a direct proof 
here using Lemmas 2.1-2.3. First, the lelnmas ilnply E(gT, 0.  +) = 0 Ulfless d >_ 0 
is an integer. \Vhen d > 0 is an integer, the lemlnas and (2.10) ilnply 

 - l H 
Ea(g-, O,  -) = q pp(d) 

1 + eq(d)2iv½pk(4rdv)e(dr) 

1 
= 4vp(d)pk(47dv)e(dr). 

(2.11) 
Tlle saine |elnmas also ilnply 
E(g-, 0.  +) = qA(1,¢)+(g,-.O) + q½M*(O)+(g-.O) 
1 1 
= hv + A(O, 
1 
= 2hv. 

This proves (2.8). 
As for (2.9), we again check terln by terre, and it is clear frolll tlle leIllllltlS that 
E'(g-, 0, -) = 0 uIfless d is an integer, which we assume froln now on. 
WheI d < O, II'd.,(9-,O,'b-) = 0 b3 Lemllla 2.3(3), and so (using Lemlnas 
2.1-2.3 and (2.10)) 
"*' ' 0 -* , 
e'(9»0,-)=qu.tg, . oe)t,a(l 0, Il 
= -2,,½q(-v)«()/1 - )) Il 

1 
= -2v - p(-d)qk(-47rdv)e(dr), 

as desired. 
VVhen d > 0 and eqd) = 1, one has II'*.q(1.O, -) =0 and 
-1 
IId,q(1,O, 4 -) -- v/(ordq d + 1) logq. 
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The saine computation using Letnnlas 2.1-2.3 and (2.10) yields 
*I -- 1 
E e (g-, 0, (I)) = -2vpk(4nd'v)e(dr)(ordqd + 1)p(d)tog q 
1 
(2.12) = -2va,pk(4ndv)e(dr), 
since a = (ordq d+ 1)p(d)logq in this case. 
When d > 0 and eq(d) = -1. there is a prinle lld such that II)*t(1,0,t ) = 
pt(d) = 0 by 22.10). In this case, 
1 (ordt d + 1) log l. 
The saine calulation yelds 
E d (g,O. -) = -2va,,p(4dv)«(d7), 

as desired. 
Finally, 

(2.13) 

with 

22.14) 

go 

when d = 0, one has by the Saille lenlnlas, 
1 
t;(gT, s, +) = k . (G(s) + G(-0) 

k 
G(s) = (qv)  A(1 + s. e) H( j + -). 
j=l 

(2.15) E('(g-, 0, (I)-)- 

"G--420) = hv½ Çlog(q.')+ 2-- 

This finishes the proof of (2.9). 

A(1,e) 

[] 

One has by Proposition 2.4. 
*(,o) lv--' * 
= E (g»O,,O +) 
2 

Proof of Theorem 0.1. 
(2.16) 
and 
(2.17) E*('r, 0) = v --½ lE *' l klE* ] 
L t..o.-)- ,=7 (-'°+) " 
Now Theorem 0.1 easily follows from Propositions 1.1 and 2.4 and Theorem 2.5. 

3. PROOF OF THEOREM 0.2 

By Proposition 1.1 and Fornmlas (2.16) and (2.17), Theorem 0.2 is equivalent 
to the identity 
()2#+ ( -1 ï)(E'(9-,0.+,' 
(/*(g_,O,5+) =i l log q I k (gr.O,ê),]" 
(3.1) E'*' - 
t,s*'(-,o,<-))  
To prove (3.1), one observes the following trivial but fmidamental identitv and 
computes both sides: 
(3.2)  --1 * 
s; ( g«,,,I,-) = s; (w<,,-,,,I,+). 
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Here wf and tt, are the images of w = (ï 1 ) in C,(Af) and G(IR) respectivelv. 
The left-hand side of this identity is given by 

Lemma 3.1. 

For the right-hand side of (3.2), Olle has 

Lemlna 3.2. 

E*'(Wfgq-,O, dP-)] =i ½1ogq E*'(g-,O,(P-)]" 

Pro@ We verify these identities by comparing the Fourier coefficients E (WfgqT, 
s, dp +) with E(9, s, dp+). We nmy assulne that d is an integer bv Lemnms 2.1-2.3. 
Straightforward calculation using the saine lenmlas yields, for any integer d, 
(3.3) IV,»(Wygq¢,S, dp +) = F(d)II'.»(gT, s, dp +) 
with 

{ ç if p { qoc, 
(3.4) F(d) = '- ifp= OE, 
_1_ 1 l+eq(d)q -st 
 l+%(d)q_(+} if p = q. 
Here r = ordq d. We will verif.v the derivative part and leave the value part to the 
reader. First assume d -fl 0. It follows rioto (3.3) that 
E*" lE*'" ! ifeq(d)=-l, 
a_ (Wfgqr, 0, (I)-) = d tg-, 0, (P-) [,i  if {[q (d) = 1. 
q ordq d+l 
When eq(d) = 1, one bas by (2.11) and (2.12), 

No 

d g-,O, dP-) _Ed(g,O,+)ordqd+ 1 logq. 

(3.5) 

= Ed (gr,O, P-) +  logqEd(gr, O, +), 
Ee_ (WfgqT, O, (--) . ,, i 
q 

as desired. When eq(d) = -1 we have E(g-, ¢0, (I)+) = 0, and (3.5) still holds. 
It remains to check the constant term. Recall (2.13)-(2.15). Direct calculation 
using Lemmas 2.1-2.3 also gives 
(3.6) E)(Wfgqr,S, dP+)=:[:  (q-G(s)+q-G(-s)). 
I]j=l (J -}- ) 
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Therefore, 

(3.7) 

E.,, , 2G'(0) .2G(0) l log q 
0 uf9qr'O'dP-) =i k--, q-Z k! 2 
i 
= iE'(g,-, 0. qb-) +  logqE(gq-.O. b+), 

a.s expected, too. 

4. L-SERIES 
Recall that q is a prime congruent to 3 modulo 4 and k = Q(v) is the 
associated imagi,mry quadratic field. Recall also ([Rob]) that a canoifical Hecke 
character of k of weight 2/," + 1 is a Hccke character p satisfying 
(1) The conductor of p is 
(2) p('2t) = It(ç() fiw an idem 1 relativcly l)rime to 
(3) t(aOk) = +o 2k+l. 
In this section, we will give an explicit forlmfla for thc central derivative of its 
L-fllnct.ion, which has dcep arithlnetic implications as mcntioned iii the introduc- 
tion. \Ve rcfer to [Gro] tbr the arithlnetics of elliptic curves associated to these 
Hccke characters (sec also [iXlY] and [Ya] and the refcrence there for more recent 
devclopments). For each ideal class C of k, we can define the partial L-series bv 
(4.) c(.,t,, c) = 
C, integral 
Of course, L(s, #) = -ce CL(k) L(s, t. C). The following proposition is standard. 
Proposition 4.1. Let  Ç Che a primitive ideal of le relatively prime to 2q. and 
u,rite 
9 = [a, b + V/ ] with a > O. b = O modq. 
9 

b+ v' 

2aq 

L(s+l,'+ 1. p. C') -- 

(4.6) 

¢(r) = ao(V) - 2 Z a,pk(4rrm')e(nr) - 2 Z p(-,,)q,.(-4rrm,)e(r-r). 

E* (r, 0) = v-O(r) 

t,("a) 
(.oE.o.l)2k+l (2v/)-L(2s + l,e)E(r,2). 

and 
Then Theorem 0.1 savs that 
and 
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Corollary 4.2. Let the notation be as in Proposition 4.1. 
(1) The central L-value is 

L'(k + 1, p., trivial) 

7r 1 i 

Pro@ Only the second one needs a little explalmtion. When (-1)k() = -1 ,ve have L(k + 1. p. C) = {1 automaticallv and thus O.(r,a) = ). So Theorem 0.1 and 
Proposition 4.1 imply 

L'(k + 1, p, C) = 

When C is trivial, one tan take  = Otc. In this case, a = 1 and 
and ttms 
çb(r,a)=çb(-+)._ 

b -- 1 lllod 1, 
2q -- 2 

In recent joint work with S. Miller ([MY]), xe proved that L'(1,p,trivial) > 0 
when q = 3 mod 8 and k = 0. Combining that with Corollary 4.2, om has the 
following curious inequality: 

(4.7) 
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SYSTEMS OF DIAGONAL DIOPHANTINE INEQUALITIES 

ERIC FREEMAN 

ABSTRACT. ,Ve treat systems of real diagonal forms Fl(X), F(x),..., FR(x) 
of degree k, in s variables. Ve give a Iower bound so(R, k), which depends only 
on R and k, such that if s >_ s0(R. k) holds, then, under certain conditions on 
the forms, and for any positive rem number e, there is a nonzero integral sinul- 
taneous solution x E Z s of the system of Diophantine inequalities IFi(x)l <  
for 1 < i < R. In particular, our result is one of the first to treat systems 
of inequalities of even degree. The result is an extension of earlier work by 
the author on quadratic forms. Also, a restriction in that work is removed. 
which enables us to now treat combined systems of Diophantine equations and 
inequatities. 

1. INTRODUCTION 
1.1. Statement of main result. In 190. Schmidt [17] proved a far-reaching 
result about systems of Diophantine inequalities of odd degree. Given anv odd 
positive integers dl,...,d, Schmidt showed that there exists a positive integer 
si = si(d1 .... ,d), depending onlv on dl,...,d, with the following property: 
given any positive integer s _> s and any rem forms, or homogeneous polynomials, 
Gl(X) ..... G(x), in s variables, of respective degrees dl, .... d, and given any 
positive nmnber , there alwavs exists a nonzero integral vector y  Z s satisfying 
the systenl 
(1.1) IGI(y)I < e, IG(y)I < e .... , IG(y)I < e. 
So, in other words, as long as the forms are ail of odd degree, and are defined 
in enough variables in terres only of the degrees, then there is a nonzero integral 
solution of the inequalities (1.1). Many particular classes of svstems of the type 
(1.1) bave been studied. 
For Diophantine inequalities of even degree, the situation is nmch different. 
There is no such general result as above for integral solutions of Diophantine in- 
equalities of even degree, and in fact there are few results at ail for inequalities of 
even degree. (However, results are known if one allows solutions in algebraic inte- 
gers in purely imaginary number fields. See Theorem 11.1 of [22].) In this article, 
we present one of the first results concerning systems of Diophantine inequalities of 
even degree, while at the saine rime removing a restriction from an earlier paper by 
the author, on quadratic Diophantine inequalities [10]. We are now able to remove 
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mo (/, &, ) = / 
and 
(1.5) 

the l'estrict.ion bv com|)ilfilig the powerful ideas of B(mtkus and G6tze [3] with the 
techniques of Nadesalingaln and Pitlnan [16], and bv adapting our previous work 
in [10] and [11] to treat the miuor arcs properly. 
To state out first result, we require SOlne notation and definitions. X shall be 
working with systems of diagonal forlns Fi(x) given by 
(1.2) Fi(x) 
For systems of forms Fi as in (1.2), we define the coefficient matrix of the system 
F to be the matrix 
(1.3) A = (AO)<i<n. 
ljs 
For 1  j  s, we denotc the jth çohnnn of A by Aj. 
Now suppose that ,1 is a subset of the set of indices {1,2 .... 
to I)c the submatrix of A consisting of the cohmms Ni with j  J, and we define 
r(A.) to 1)e the tank of the matrix Aj. Finally, if x  N  satisfies Fi(x) = 0 for 
1 < i < R and the matrix 
lSjSs 
is of full tank, then we sav that x is a nonsingular solution of the system F. 
Now, for integers R and k and any rem nulnber u. we define the flmctions 
(.4) 
rein (4R  + 4R + 1,384 log 16R + 5) if k = 2 
Rk log 2k 
lg2 + uklog(Rlog2k) if k is odd and k  3 
h [48k  log 3Rk 2] if k  3. 

{ ( 5 ) 
no(k,u) = nfin 2k-t - 1, k(logh+loglogk+2)+ ukloglogk 
log k 
V'e can now state out first result. 

if k = 2 
if k>3. 

Theoreln 1.1. Suppose that k is an inte9er with k >_ 2 and that R is a positive 
integer. There are absolute real positive constants C and C2 for which the following 
property holds: 
Suppose that g is an integer satisfying 
(1.6) « lllgX(?/,0 (/?,',dl) ,N0 (]x'.d2)) . 
Suppose that s is an integer satisfyin9 s >_ gR. For 1 < i < R. suppose that Fi(x) 
is a real diagonal form of de9ree k, as in (1.2). Let A be the coefficient matrix of 
the system F, as in (1.3). Assume that the followin 9 two conditions are satisfied: 
(i) Either k is odd, or there exists a real nonsin9ular solution y of the system 
FI(y) = F2(y) ----- .... F/¢(y) ----- 0. 
(ii) For every subset J C_ {1,2 ..... s}. one hs lJl <_s-e(R-r(Aj)). 
Fix any positive real number e. Then there is a nonzero inte9ral solution x G Z  of 
the system 
(1.7) IF(x)l <  for 1 < i < R. 
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For general even/% Theoreln 1.1 is one of the first results of its kind to our knowl- 
edge. We note that at least somc conditions similar to Ci) and (ii) are necessary, as 
may be seen by considering the examples given after Theorem 2 of [7]. 
For odd/,', we note that Theorem 1.1 is hot very much of an improvement be.vond 
that given bv Nadesalingam and Pitman [16], and could presumably be obtailmd 
by combining their methods with results of Vaughan [19], [20] a.nd work of Woolev 
[23]. We also observe that, following the method of Section 7.2 of [16], we could 
relnove condition (ii) for odd/," if we chose to do so. 
As well, our lnethod of proof shows that mlder the conditions of Theorem 1.1, ve 
can give a lower 1)ound of the expected order of magnitude, P-, for the nmnber 
of solutions of (1.7) in a box of size P. ff»r all sufficiently large P. This was hot 
previously knowm even in the special case of systelns of inequalities of odd degree. 
We emphasize that when using our methods, condition (ii) is necessary to obtain 
this lower bound. Presulnably, one could also give an as3mptotic formula for the 
nulnber of solutions, bv combining with the methods of [12]. 
Note that we bave ex«luded the case/," = 1 from the statenlent of the theorem. In 
this case, oto" knowledgc is ranch l»etter. For h = 1. if one has 8 >_ /ï' -1- 1, a. nonzero 
solution of (1.7) may bc follnd using a l»ox principle, whcther or hot condition (ii) 
holds. (Sec the Lemma in [4].) For k = 1. one can also find many solutions of (1.7) 
in a box of size P. (Sec Lemma 1 of [9].) 

1.2. Combined systems of Diophantine equatiolls and inequalities. \Ve 
note now that it would actuallv bc faMv routine to give at lcast ont result on 
inequalities of even degrec; one could simply generalize the work in [10] by con> 
bining those techniques with the methods in [11]. However. such a generalization 
would exclude many important classes of svstems of inequalities, fol" example those 
in which some of the forms are integral. We were forced to exclude such systems 
iii [10] because of the methods we used. In out current work, we are able to treat 
these formerly excluded systems. We give some background to more full,v expla.in. 
In [10], we considered simultaneous systems of diagonal quadratic Diophantine 
inequalities. Fol" a positive integer R. definc, for 1 _< i _< R, the real quadratic 
forms 
Q(x) = a,î + a2«g +... + a,«. 
It was proved in [10] that for every positive real lmmber e, under certain conditions 
on the system of forms Q, Q_ ..... Qn, there is an integral vector x  Z  \ {0} such 
that one has 
In that paper, one of the conditions we assumed was the following. (Sec condition 
(iii) of Theorem 3 of [10].) 

(1.8) 

For each choice of (fit,/32 ..... 3n) C liRn \ {0}, there is at least 
one coefficient of fltQt + f12Q2 + ... +/3nQn that is irrational. 

This condition allowed us to use a modification of the remarkable work of Bentkus 
and G6tze [3], but excludes certain important systems ff'oto consideration. The 
restriction (1.8) rules out systelns in which one or lnore of the forms is an integral 
form, and also any system iii which any nontrivial linear combination of the forlns 
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is a.n integral form. The centra.l new contribution of this paper is in removing the 
condition (1.8). 
We now state a lnore technical and lnore genera.l version of Theorem 1.1. We 
require lnore nota.tion. For a. rem vector  = (/31,...,/3n) E N u and a system G of 
forms G1 (x), G2(x) ..... Gn(x), we define the forin 
(t-G)(x) =/C (x) + 2C(x) + ... + 
Also, for rem nulnbers z, wc define 
e() = e2. 
Theorem 1.2. Suppose that k is an integer with k >_ 2 and that r and R are 
integers with R >_ 1 and 0 <_ r <_ R. Then there are absolute positive real constants 
C and C with the following propertg: 
Deflne no (k, u) as in (1.5). Suppose that f is an integer satisfying 
(1.9) « >_ ,,0 (k,02) . 
Let s be an integer witb s >_ fR. Ah'o suppose, for 1 < i < H. tbat 
(x) = a,l.-î + a: +... + 
is a diagonal form witb real coefficients. Let ,4 be the coefficient matmx of the 
system F, as in (1.3). 
Assume that the following four conditions are satzsfled: 
(i) Either k is odd. or tlere exists a real nonsingular solution y of the system 
Fl(y) = F(y) ..... Fn(y) = 0. 
(ii) For every subset J ç {1.2 .... ,s}. ov bas [J[ _< s-f(R- r(Ar)). 
(iii) Tbe forms F, F ..... Ff bave integer coefficients: also. if  = (a. aee .... , 
oR)  IR n and o  F is a rational form, tben ar+ = ct+2 ..... an = 0. 
(iv) If r >_ 1 bolds, tben there is a positive real constant c(F) suct that one has 

q=l a:(a ..... ar,q)=l 3=1 oe:l i=1 
l <_ai <_q (l_<i_<r) 

Fix any positive real number e. Then there s a nonzero integral solution x  Z  of 
the system 
Fi(y) = 0 for l<i<r, 
(1.10) 
IFi(y)l < e for r+l < i < R. 
Moreover, if we define m0 (r. k, u) as in (1.4) avd we assume that tbe condition 
e  II 0 (r, k, d 1) holds, then we may omit condition 
from 
OUF 
assumptions. 
Some discussion o( the condition (iii) is warranted here, since it is the most 
important distinction between Theorems 1.1 and 1.2. In Theorem 1.1, we consider 
systems of inequalities (1.7). Now, if one chooses e < 1. and F(x). sa.',', is actually 
an integral form, then the system (1.7) reduces to the system 
t:(y) = 0. 
IFi(y)l<e for 2<i<R. 
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So, in this case, the systeln of R inequalities aetually reduces to a systeln of one 
equation and R- 1 inequalities. A similar reduction occurs if some nontrivial real 
linear Colnbilation of the forlns ri, soEv ctF1 + c'.)F2 + ... + aRFR, is an integral 
fornl. In these situations, one might say that there is açtually ail equation hidden 
iii the system of inequalities. The conditi(m (iii) ensures that there are actually r 
equations in the systeln and R - r "truc" inequalities. It is helpflll to ensure that 
there are hOt any lnore "hidden" equations becmlse it turns out that one requires 
more variables to treat the system if there are more equations present. One lna.v 
think of Theorem 1.2, more or less, as the sub-case of Theorem 1.1 in which there 
are exactly r equations present in a systeln of /? inequalities. \Ve note that the 
second clause of condition (iii) is vacuous if r = R holds. 
One might question if, in condition (iii), the terln  -F could be replaced by 
the terre aï+Fï+ + c+2F+2 + ... + cRFR, to give a slightly weaker condition 
asserting that r equations are "'hiddelf" in the systeln. It turns out that one tan 
not, as ma.v be seen by considering the example 

F(x) = 
F(x) = 

k 
"]['8 ' 

where a 3, Ci4,..., as are anv integers. Here, for r = 1 a.nd R = 2, COl(lition (iii) does 
hOt hold, since (1/x/) (F2 - F1) is an integral forln, filial thlls tac systenl (1.10) is 
equivalent iii this case, for Slnall e, to the s.vsteln 

1 
Fl(x) =  (F:(x) - Fl(X)) = 0; 

on the other hand, for any nonzero real nulnber a2, the forln ct2F 2 is not a rational 
forlnsincetheratio ((3+v)/(2+v/-))isirrational:sothissystelndoesnot 
satisfy the suggested replacement condition. So the putative replacelnent condition 
is hOt strong enough. 
We now discuss condition (iv). The tenn  is the so-called singular sertes, 
and condition (iv) silnply states that it is bounded below by a positive constant, 
a necessary precondition when using the Hardy-Littlewood method. As the last 
sentence of Theoreln 1.2 states, we could have omitted the condition from our 
assumptions iii favor of a lower bound for /. However, since the consideration of 
the singular sertes is not our central lotus in this work, we have chosen to include 
condition (iv) so that out result can be improved immediately and transparently 
if improvements arise concerning the singular sertes and the p-adic problem. This 
should certainly be possible in the case k = 2, for example. Also, we wish to clearly 
indicate that tUe condition /? _> m0 (r,k,C) is needed Olfly because of tUe p-adic 
problem. 

1.3. Related results. çVe now compare out work with other results. For even k, 
Theorem 1.1 is an analogue of a result of Davenport and Lewis, coucerning systelns 
of Diophantine equations of even degree. (See Theoreln 2 of [7].) They assmne that 
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a svstem of diagonal equations 
FI(x) = F2(x) ..... FR(x) = 0. 
of even degree k. with k > 2, has a real nonsingular solution and also that foi" 
1 _< S _< R, every set of S independent integral linear combinations of Fa ..... FR 
contains at least 
(1.11) [4RSk 2 log (3Rk2)] 
variables that aI)p«'ar explicitly. Under these conditions, the system of equations 
bas a nonzero integral solution. \Ve note that if one replaces the quantity in (1.11) 
bv Sé, and restricts to integral forms, then one can show that their second condition 
is equivalent to condition (ii) of Theorem 1,1. 
Nadesalingam and Pitman [16] proved that any R real diagonal Diophantine 
inequalities of odd degree k, with k _> 13, iii .s variables with 
. > 3Rk 2 log (3Rk) 
bave a nonzero solution. We note that thev do hot require anv condition that is 
sinfilar to condition (il) of Theowm 1.1. Also, we observe that they could certainly 
have used their mothods to obtain similar results, although with a different lower 
bomd foi" s, in the cases k < 13, but iii order to streamline the presentation thev 
did hot do so. 
Finally, we note that Briideln and Cook [5] bave given a lesult on systems of 
diagonal Diophantine inequalities of odd degree. Under certain conditions on the 
coefficient matrix of the system, they show that there is a nonzero solution of 
the system of inequalities. They require an assmnption similar to condition (ii) 
of Theorem 1.1 and also a condition that is stronger than (1.8). The lmmber of 
variables they require is on the order of Rn0 (k. C2). We also note that 
they 
call 
find a lower bomd of the expected order of inagnitude foi" the mnnber of solutions 
of their system iii a box of size t » for a sequence of positive P tending to infinity. 
although hot for all large P. as out treatment provides. 

1.4. Methods used. The general strategy of the proof is to combine the method 
of I3entkus and G6tze [3], which is very effective for Diophantine inequalities, with 
the techniques that Nadesalingam and Pitman [16] use to treat combined svs- 
teins of Diophantine equations and inequalities. We remark that the techniques of 
Nadesalingam and Pitman are themselves a combination of the Hardy-Littlewood 
method and the Davenport-Heilbronn method. Using the techniques of Nade- 
salingam and Pitmm allows us to treat those systems of inequalities that contain 
'qliddeff" equations. Fol" those who are fmniliar with their argmnent, we note that 
we do hot have a so-called residual set in our proof, as in their paper. 
One other crucial result is needed, and this involves showing that, on the minor 
arcs, oui" exponential sulns are smaller than the trivial bound. Iii previous work 
on these t,vpes of problems, including [10], [111, and essentially also [3], such a 
result was achieved by splitting the minor arcs into two regions and handling each 
separately. In this paper, we handle both of these regions together, which is hot 
onlv cleaner, but also seems to be necessary here. 
I would like to thank Scott Parsell foi" showing me how to improve Lemma 6.1. 
I would also like to thank Michael Knapp and Professor Woolev for indicating to 
me how to prove part of Lemma 8.5. 
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'2. DEDIÇTION OF THEOIEM 1.1 FFIOlkl THEOIEM 1.2 
The lmlk of this pal»er is dedicated te proving Theoreln 1.2. In this section. 
however, we delnonstrate how Theoreln 1.2 implies Theoreln 1.1. Te this end, we 
consider a system le of real diagonal ferres F1, Fa,..., FR as in Theorenl 1.1. 
We give a definition first. Suppose tlTat G1, G9 ..... GR and lit, ici.,_ ..... li/ are 
two svstenls of forlns. If there exists a set of R lilmarly independent real vectors 
/3, fie ..... /3R E IR R such that 
H(x)=/3-G fir 1 <i< R, 
then we say that the svstem H is equivalent te the system G, whi«h we denote by 
G  H. It is easv te check thaI this is in fa«t an equivalence relation. \Ve olserve 
as well that if G is, in particular, a system of diagonal fol'mS, and G , H holds. 
then H is also a system of diagonal ferres. 
For anv system of ferres G. we define z(G) te be the lmml)er of ferres among 
GI,G_ ..... Gn that are integral, that is, whose coeffi«ients are all integers. New 
for out systent of ferres F, we define 
r = r(F) = ma.x 
In othcr words, r(F) is shllply the lllaXillllllll llllllll)er of forlllS that are integral in 
allV systenl G equivalent te F. \Ve clearlv have 0 _< r _< B. 
New suppose that G is a system equivalent te F and that G has r integral ferres. 
Se there exist H real linearly indcpendent vectors/3./3e,...,/3/ E IR R such that 
Gi =/3, - F for 1 _< i _< R, and also the systeln G contains r integral ferres. Bv 
relabeling if necessary, we mav aSSllllle that G1, G2 ..... Gr are integral forlllS. V,e 
new show that conditions (i)-{iv) of Theorenl 1.2 hold for this systenl Cf. Then 
we will apply Theorenl 1.'2 te Cf. and we will see that the nonzero sohttion of the 
systenl G is also, under certain Coliditions, a solution of the svstelll F. 
Silice F is equivalent te Cf. if thc coefficient lnatrix of F is .4. then the coeflï«ient 
matrix of Cl is TA for the nonsingular R × R ulatrix T with rows/3,. Thus. for any 
subset J C_ { 1, 2 .... , s}, we have 
(2.1) r((TA)s) = r (TA.,) = r(As). 

The systenl F has a nonsingular sohltion 
{1.2 .... ,s} v«ith IJI = H satisfying r(Aa) 

x if and onlv if there is a sul)set ,1 of 
R and H md -j: 0. Thus the existence 
jd 

of a real nOlsingtllar solution for G follows ff'oto the existence of such a sohltion for 
F. Se condition (i) holds for G. By (2.1), it is easy te see that condition (ii) holds 
for the coeflïicient lnatrix of G. becallse it holds fol the coeflïicient lnatrix of F. 
New we turn te showing that condition (iii) of Theorem 1.2 holds for the system 
' .... a)  IR R is a real ve«tor such that 
G. Te this end, suppose that et' = (a, au 
! i 
et'- G is a rational fonn. We need te show that o,.+1 = Or+ 2 .... = O R = 0 holds. 
This holds vacuouslv if r = R. For r < R. clearing denonlinators, we see that there 
is a nonzero integer n such that. defining et = et', we have 
et-G = .et'-G  Z[x]. 
Since  is llOllzero, to prove that a+ 1¢ = a+ 2¢ .... - = 0 Re = 0 holds, it is enough 
to prove that we have c+l = Ctr+2 ..... OR = 0. 
So SUl)pose that this is hot the case. Lett.ing el, e ..... eR 1)e the standard relit 
basis for IR R, we then have that e.e2 ..... er. et are r+ 1 lilmarly ilMependent 
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vectors in R. çVe inay thus extend this set to a basis, sa.,," 9',9', .... 3' of 
N n, with i = ci for 1  i  r, and r+l = " Then the system of forms 
1 " G, 2 " G .... ,n - G is equiwdent to G, and thus in turn to F. But its first 
r + 1 forms are integral. This contradicts the definition of r(F), and thus we lnust 
in fact have 
-r+l  r+2  "'"  R  0. 
Thus condition (iii) holds for G. 
since  satisfies (1.6), wP havÇ g oE m0 (,k, Çl). alld tlms we certainlv 
Now, 
have g  mo (r, k, Ç). whence by the final sentence of Theorem 1.2, condition (iv) 
is UlmeCeSSaly. Since we have also secn that conditions (i)-(iii) of Theorem 1.2 hold. 
we lnay apply Theoreln 1.2 to the system G. 
Before doing so, we give some more notation. For a vector x = (z, w ..... x)  
N , define 

Ixl = ,l,aX Ixl. 
l<j<s 
For an R x s matrix BI = (tlij)l<i<R, We define 

l<j<s 
We note that the notation differs slightly from that used b5 some other authors, 
for exalnple Nadesalingam and Pitlnan [16]. Similarly, for a system F as in (1.2), 
we define 

IIFII = II/'XR Iijl  

Defining the matrix T as above, we certainly have det(T) ¢ 0 and I[Tll ¢ 0: 
so we may apply Theorem 1.2 to the system G with e replaced by the quantity 
(Idet(T)l¢) / (RIITII-). X obtain a nonzero integral solution x e Z s of the 
system 
Idet(T)l 
la(x)l < (1 < i < ). 
IITII -1 
By Çramer's rule and Hadalnard's rule, iI follows that x is also a solution of the 
systeln (1.7), whence Theoreln 1.1 follows. 
çç now turn to the proof of Theoreln 1.2, which colnprises the rest of the paper. 

3. INITIAL ]EDUCTIONS 

In this section, we reduce the problem of proving Theorem 1.2 to the consider- 
ation of a systeln of forms as in Theorem 1.2. but under a few more restrictions. 
This will lnake our application of the Hardy-Littlewood method casier. We first 
note that by considering the forms e-lFi, it is enough to consider only the case 

çVe can also assume that we have 

IIFII > 1. 
If this were not the case, then (1.0.0 ..... O) would be a solution of the system 
(1.10) and we would be done. 
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We now quote a lenmia, which seems to have been first used in this field bv Low, 
Pitmau and Wolff. (Sec Lemma 1 of [13].) It is actually a special case of a result 
on matroids, apparently due originally to Edmonds [8]. A proof tan also be fotmd 
in Aigner. (Sec Proposition 6.45 of [1].) 
Lelnlna 3.1. Let A be an Rxs "matrix over a field K and let w be a positive iuteger. 
The matrix A bas a R x Rt, pa, rtitiomble s'ubmatriz (that is. A icludes w disjoint 
Rx R submatrzces that are to,sitgula, r over K ) if a,td o'nlg if the followin 9 co,tditiot 
is satisfied: 
(3.1) [J[  s - u,(R- r(Aj)) for all sub.s'ets J ç {1.'2 ...... }. 

To be clear, by including w disjoint R × R nonsingular suhmatrices, we mean 
that there is solne pernmtation of the cohmms so that the first R cohmms form a 
nonsingular matrix, as do the second R colunms, and so on. 
Note that the condition (3.1) is exactlv condition (il) of Theorem 1.2 in thc case 
« = (. Thus we may apply the lcmma to Ihe coefficient matrix A of the svstem F, 
with the choice u' = (. Thercfore, A has ill R X ( partitionable sulmiatrix. By 
relabeling vmial)les if ne«ssary, we lnay write 
(3.2) A=[A A .... Ce Acn+ A«n+2 ... A, ], 
where A. is an R x R submatrix for 1 N v N g and where 
(a.a) A.=dt(A,,)l¢0 for lvSC. 
Now considcr the system F in the case that k is odd.  show that F has a real 
nonsingular sohltion. Since A has the form (3.2), and (3.3) holds, one can sec that 
F is equivalent to a system G with coefficient matrix B such that the left-hand 
R x 2R sulmmtrix of B has the form 
here I is the R x R identity matrix, and B.2 is a nonsingular R x R matrix.  can 
find real nunlbers z+a, z+2,..., z2 satis-ing 
zR+ [ - 1 
ZR+ 2  -1 
2 = . 
z2 -1 
Nowlet zj = 1 for 1  j  R, and let zj = 0 forj > 2R. Then for 1  j  s, 
define z = zj , which is always real, since k is odd. Setting x = (Zl,Z2 ..... 
one can observe that Gi(x) = 0 for 1 < i < R. Now the left-hand R x R matrix of 
-(OG has detenninant " -1 -1 -1 which is nonzero. Thus the 
h"   " " " Y 
system G has a real nonsingular solution, whence, as in Section 2. the svstem F 
does as well. 
Thus, whether k is odd or even, we know that there is a nonsingular solution of 
the system 
F(x)=0 (1 <i<). 
 now show that there is a real nonsingular solution whose conlponents are all 
positive. As noted above, there is a subset J = {j,j2 ..... j} Ç {1, 2 ..... s} with 
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]J] = R and a real vector x E ]s suçh that we have det(Aj) ¢ 0 and H xj ¢ O. 
Noxv, tbr 1 < i < R, we define the linear fornl 
(3.4) Li(y) = Li(y,Y2 .... ,Ys) =  AijYj. 
j:l 
 » for 1 < j < s, we see that th('re is a rem vector z = (z ..... z)  
O11 setting zj = ai _ _ 
R  such that zj ¢ 11 holds fl»rj  .1, and we have 
(3.5) L.i(z)=0 for 1 <i<R. 
Nov, if k is even. oto choice of z ensures that we have zj  0 for 1  j  s. If 
k is odd, thon for each j, we lllay if necessary replace zj by -zj, and replace the 
('oeEicients ij by --/kij for 1  i  , and consider the resulting system, hl this 
lllilllllel', Wç IIIHV ellSllre that we bave a solution z of (3.5) with zj  (I for 1  j  s 
and zj > (I for j E ,I. Note that Colditions (il) and (iii) of Theoreln 1.2 and 
«onditims (3.2) and (3.3) are mmffe«tcd. Con(litiol (iv) is also unaffected, since 
q ("  tl'l'k  zjfl.) 
the Slllll ,Ç is always rea.1 for odd k with A O  E, which mav be 
J'=[ fil 
scen lv subsfimhg -« for m h tire SUln. 
Now suppose tha z = 0 for some j0 safisfying 1  j0  s. %%% clearlv bave 
fix a positive rem mmlwr ? wih 
nllll  
Shce Ad is nonsingular, thee is  rem vtor w = (w,..., teR) such that we lmve 
A.w = -Q Aço. 
Bv Crmner's rule md Hadmnmd's rule, we certinlv hve 
I'1 -< Id«,(A,)l <-  m. 1 o t <  < . 

Now defilm 

,  zj + tt'i 

forj =ji E J 
for j = jo 
fol" j ¢ J LA {jo }. 

Writing z' ( z'. "' 
= -2 ..... Z), we have 
Li(z t) = Li(z) =0 fol" 1 <i < . 
Also, we hae ' 
 > 0 for j  JU{j0}. All ofthe other COllpOlWnts ofe' are 
equal to the respective COlnpOlmnts of e: so we bave replaced our rem nonsingular 
solution by a rem nonsingular solution that bas Olm more positive COlnpolmnt and 
tlmt still satisfios tlm çondition zj > 0 for j  3. poating this proçss as manv as 
(s - R) rimes, we can find a lonsingular real solution e = (z ..... z) with z > 0 
for ljs. 
Thus, scaling if necessal% ve lnaV choose a l'eal lmlnber  and rem munbers 
z, ï2 ..... c, tbat satisfv 
t for 1 <j<s, and 
0<Nz5 _ _ 
(.) 
L(e)=0 for l<i<R. 
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To sure up, iii this section, we have delnonstrated that to prove Theorem 1.2, it 
is enough to consider a system of forms F1, F2,..., F/ as iii Theorem 1.2, with the 
added assumptions that IIFII > 1 hohts, that thc coefficient matrix .4 of the system 
satisfies the conditions (3.2) and (3.3), and that there is a real ve('tor z satisf,ving 
(3.6). Iii seçtions 4 9, we l)rove Theoreni 1.2 mlder these ad(titional assmnl)tions. 

4. THE DAVENPOIRT-HEILBRONN IIETtlOD" TttE ,¢";ETI:P 
Now we proceed with the proof of Theorenl 1.2 under the additional assmnptions 
we ruade a.bove. ' shall essentially use the Hardy-Littlewood method, in an 
involved form. X> combilie the methods of Bentkus and G6tze [3] with those of 
Nadesalingam and Pitlnan [16]. 
 note that thl'oughout the pal»er, implicil constants in the notation o() and 
O() and << a.nd >> lllay dot)end on R. s, k, , t, the coecients of the tllllS F1 ..... 
F, and the real ve«tor z. 
> cousider the lmmber of solutions of the svstem 
(4.i) F,.(y) = 0 f,,r 1 < i < ,', 
IF(y)l 
In the usual fashion, we use a real-valued, even kerncl flm('tion K  N + N to give 
a lower bomM for the nmnber of integral solutions of the svsWm {4.1) in a certain 
range. Define such a flm«tion K, fi)r any real mlmber a,, bv 
., 
By Lemma 14.1 of [2], fi)r a.ny real nmnber u, the function K satisfies the identity 
] { ,, ,f ,,, 
(4.3) ¢(u) = «(,,)K(/)S = I -I"1 if I1 < 1. 
The function K satisfies, for real nmnbel'S , the bomd 
(4.4) 
% will a.lso use the identitv 
t {1 i,,: 
(4.5) (')'« = 0 if ,, e z  {0}. 
Now for positive real numbers P and Q satisfying Q  P, we define the so-called 
Q-smooth lmmbers tobe the set 
A(P, Q) = {x 
Fix a positive real number q, to be chosen later, so that it will satisfy the require- 
ments of Lemmas 5.5 and 6.1. Then for real numbers a and P with P > 1, we 
define the exponential sure g{a) over the smooth immbers bv 
(4.6) g(a)=g(a,P)=  e (a.r). 
x(P, Pv 
Also, for 1  j  s, and real veçtors a G , we define the lillear forlns 
(4.7) 
i=1 
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Then we also define, for o G I t and rem nmnbers P with P _> 1, aud for 1 _< j _< s, 

the flnctious 

gj(o 0 = gj(o, P) = g (Aj(o), P) . 

(o.s) 
We define as well 
n" = [0, 11  x R-. 
Now let .lU(P) be the number of solutions of the systenl (4.1) with xj E  (P, P) 
for 1  j  s. By using the property (4.3) of the fimction K(a) and the identitv 
(4.5), one can sec that we have 
xj A(P,P n) i:l i=r+l 
ljs 
observe that this last renmrk is justified by the fact that the integral converges 
absolutcly, which follows froll (4.4), whence we mav write the integral  a product 
of R integrals. By pulling the sums into the integral, we m" rewrite the above 
bound in the form 
s R 
(4.9) 
I" j=l i=r+l 
Thus, to prove Theorem 1.2. it is enough to show that the right-hand side of (4.9) 
is at least 2. 
To this end, we give a dissection of the region of integration Il" into three subsets. 
oughly speakiug, we expect that the main contribution to the integral in (4.9) 
cornes from the region where the first r colnponeltS of  are "close" to rationM 
lmmbers with slnall denolnilmtors and the lt R - r components of  are very 
small in absolute value. We will shov that the contribution to the integral in (4.9) 
from this region, the so-called major arcs, is positive and "large", and we will also 
show that the contribution to the integral froln the other regions is slnaller, and 
thus the integral over all of Il" is positive. 
For notational ease, we set 
1 
(4.m)  = 
4(n + 1)" 
We now define, for positive integers q and integral vectors a = (al, a2 ..... af) G 
Z r, and real nmnbers P with P  2, the region ,M(q, a), or (q. a. P), bv 
(q,a) = {G[0,1]  x [-(logp)Bp-k,(logp)Bp-k]-. 
i  (logp)Bp - for 1 < i < r  
q 
(4.11) 
here I]xl[ denotes the distance from the real number x to the nearest integer. X 
define the nmjor arcs to be the region 
(«.12) =(P)=   (q,,p), 
lq(log p)B a (mod q) 
(a ..... a ,q)=l 
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where by a (mod q) we mean that a nms over vectors a E Z r such that one has 
1 aiq for 1 < i<r. 
In section 5, we will prove the existence of a functiol T(P), with T(P) k 1 and 
limp T(P) = oe, and satisfyiug a certain property. The flmction will depend on 
B and the coefficients of the forms Fi. We define the minor arcs to be the region 
(4.13) m = re(P) = ([0,1]  x [-T(P),T(P)I -) N (P). 
Finally, we define the trivial arcs to be the set 
(4.14) t= t(P) = {a e  I1 > T(P)}. 

5. AN ANALOGUE OF I, VEYL'S INEQUALITY 
In this section, we give an mmlogue of Weyl's ineqnality. For any rem number T 
with T _> 1, define the region 
(5.1) mE=roT(P)= ([0.11 r X [-T.T] R-r) \.Ad(P). 
We now state the central lenmm of this section. 

Lemma 5.1. Fix a positive real number T with T > 1. Define the forms Fi(x) as 
in (1.2) for 1 < i < B. the regio, roT(P) as above, and gj(. P) for 1  j  s as in 
(4.8). Suppose that the coeciet matrix A associated with the system F bas rank 
R. Suppose also that the irrationality coditio (iii) of Theorem 1.2 holds. Then 
one bas 
(5.2) lira sup H= 'gj(.P)' = O. 
Poe «mT(P) ps 

Observe that trivially one has H IgJ(a" P)] <- P" so we are only seeking a slight 
j=l 
improvenmnt over the trivial bound. We also note that the central ideas of the proof 
stem from the work of Bentkus and G6tze [3]. 
In order to prove Lemnm 5.1, we first need to give another lemma, which is 
essentially a combination of two analogues of Weyl's inequality for exponential 
sunls over smooth mlmbers. We first quote these two analogues, essentially due to 
Vaughan and Wooley. as they are presented in [5] as Lenmms 3 and 4. respectively. 
Lemma 5.2. Let c and P be real numbers with P >_ 2. Define g(c) = g(c. P) as in 
(4.6). Fix a positive real number e. Then for su]ficiently small q. there is a positive 
real number3, that depends only on k such that either one bas Ig(a, P)[ <_ p-'r, or 
there are relatively prime itegers a ad q with q > 1 that satisfy 
g(a,P) « q¢P (q + pklqa--al) -W(2k) (logp)3. 

Lemma 5.3. Let c and P be real numbers wzth P > 3. Define g(c 0 = g(c. P) as 
in (4.6) with 0 < r/< 1/2. Fix positive real numbers A and e. Suppose that a and 
q are relatively prime integers with 1 <_ q <_ (logP) A and Iqc - ai <_ (logp)Ap -k. 
Th.en one has 
g(c,P) «, qP (q + Plqc - al) -W . 
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We now state the COlnbinatiol of these lmmnas. 
Lemlna 5.4. Define 7 = 7(k) as in Lemma 5.2. Fi.r positive real wumbers 0 and 
B'. Suppoe that P is a real umber with P  3. a,d that It is a al umber with 
Dee 9() = 9(o, P) as m (4.6), u,ith q s.uciedl small, ad suppose that oe 
(514) 19(,. P)I  ,P. 
Then lhere ei.sl. a positive itleger q and an ileger a wilh (a. q) = ] and 
q ççB',k,o I -k-kO and q - al ççB',k,O p-k-kop-. 

Proof. lt. is «lea.rly enough to aSsulno that we have 0 _< 1/2. \Ve apply Lellnlla 5.2 
with the «hoi«e « = 0/(2k). By (5.3) and (5.4), thele exist relatively prime integers 
ri ami q with q  1 su«h that t)ll(' has 
I,P 5 19(. P)I « qO/(2} p (q + pklq  _ al)-l/(2) (log p)3. 
It follows that 
q-e « p--2k(logp)6 and P]qo - a I « qe#-2k(logp)6k. 
By (5.3) and thc condition 0  1/2, we Cel'tainlv have 
('+a) 
q << (log e) qïç;) ,,« Iq - cri « (log )  -. 
Now we may apply Lemma 5.3. for large P. choosing A = 5k (B' + 3)/(1 - 0). 
say. and e = 0/(2k). XX obtain 
, « ¢/( ( + elv _ 1)-'/ 
It follows that one has 
q « p-kqe/2 and pk]q _ ai « p-kqe/2. 
Thus, since p  1 lnust hold. xve have 
2k ]l_k_k O . 
q «  2-o << and 
Thus the proof of Lcnmm 5.4 is complete. 
Now xvc are able to give the proof of Lemma 5.1. 
Proof. Suppose for the sake of contradiction that the condition (5.2) doe hot hold. 
Then there exist a positive real number e, an increasing seqnence of positive real 
mmbers P with lim. P = , and a sequence of real vectors ,  mr(P) 
with 
 nmv clearly ume that we bave e < 1. Bv trivial estimates, we have 
I(a, )l > 
Now we apply Lemma 5.4 to the sums gj(, P) =g (A)(), P) for ail suffi- 
cientlv large choices of n. For sufficientlv large t, wc have the bomds ç  P, and 
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e > (log P,)-, and also P, > 3. Thus we nlay apply Lenmm. 5.4 with p = e and 
 = 1/k and B' = 1. Therefore, there are constants çl and c2 that depend only on 
k such that for large n an(t for 1  j  s, there are integers q,a and anj that satisfy 
(5.5) 1  qnj  C1 -k-1 and [Ai(a)qi -- anj[  C2e-k-lp2k. 
It follows ffoto these bomds and the definition (5.1) of mT(P) that we bave 
for ail j with 1  j  s, and all large n. 
For fixed e and T, we thus have that la.il and q,j are uniformly bounded. So 
there are onlv finitelv manv possible (2s)-tuples 
( qnl, qn2 ..... qns, anl, an2 ..... Ons ). 
Therefore one such (2.s)-tuple, say (ql ..... q.,a ..... a). oc('urs infinitelv often. 
Thus there is some sul)sequence, say {.f,}, wi/h 
(q  .... , qa,,s, ar,,, .... , (lfims) = (ql,---, q.ç, (il .... , 
forallmZ +. 
Since the sequence {ar,} is ('ontai,,ed within the compact set [0.1]x 
[-T, TI -, t,,«r i  fm-thcr subsequence {a,,} and a ve('t,)r a0 e [0.1]x 
[-T,T] n- such that 
Our goal in the remainder of the lemma is to show that for suciently large values 
of m, we bave a,m  (P)" which contradicts our original assumption. 
By (5.5) and the defining propcrty of the subsequence {h}. we have 
(5.6) IAj(a,)qj-aj<ce--lP - for l<j<s andforall mZ + 
Taking the limit of both sidcs of (5.6) as m goes to infinity, we obtain 
(5.7) Aj(a0)= a for 1 j s. 
qç 
Because condition (iii) of Theorem 1.2 holds, denoting a0 = (ara. a0,..., a0n) we 
nmst have 

(5.8) 
Therefore we bave 

O0(r÷l)  O0(r÷2) .... = OOR. 

A,-)(o0) __ __ai for 1 
qj 
Now, by (5.6) and (5.7), we have 
(5.9) [Aj (c, - 
= - -- + -- - A(o) < 
qj qj 
for 1  j  s and for all m  N+. Now, because A has full tank, we mav assume 
by relabeling variables if necessary that the submatrix A, defined as in (3.2). is 
nonsingular. Because of this and because the bound (5.9) hokls, in particular. 
for 1 < j < R, we must have  -0 < c3(F)e -k-lP-k for some constant 
c3 = c3(F) and for all m  N+. Therefore, by (5.8). we must have 
.10 , e [0.]   [_--l2,î,--,p--  o ,,, e z+. 
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If r = 0 holds, then for m suiïicientlv large, onenmst have ,m E M(Pn). But this 
contradicts our original assulnptiol that the sequence , satisfies , G my(P,). 
whence the equality (5.2) must hold. 
So we may assume for the remainder of the proof that 0 < r  R holds. Then, 
using (5.6) and (5.10), for m ff Z + and for 1  j  s, we have 
A(.r)/ )_ al Aj(n) ad  ija.ni (( e_k_lp_ k 
(5.11) 
3  'tre qJ qJ 
Since «41 iS nonsingular, there is an r x r SUblnatrix, say A0, of A that is nonsingu- 
la.t.  assume for ese of notation that Ao is the upper left-hand r x r subnlatrix 
of AI, noting that the other cses ail follow in the saine fashion as this case. For 
any real vector a = (al,... ,an), write a' = (a ..... a,.). By (5.11), we bave 
ol/ql [ U'ml 
T  a2/Ç2 
Or/qr tt'mr 
fbr some real vector w, = (U'l. tt',) with Iwl EE e-k-lP -k Since we have 
assulned that A0 is nonsingular, we mav use Calner's rule to find b = (bi ..... br) 
with 
a/ql ] 
T a2/q2 
A 0 b = 
ar/qr 
Since A h integral entries, Olm moEv see that bi has the forln b = di/q for 
i < i < r, where dt is an integer, and q is a positive integer that satisfies 
(5.12) q  (qlq...q)det.(A)  c4(F)e -(+1, 
where the last bound follows from (5.5). 
 moEv aSSUlne, by reducing if necessary, that we have (dt, d2 ..... d. q) = I. 
By Cramer's rule again, we Inay find v  N with Av = wm, where we bave 
(s.a) lvml  «(V) -- 
Xit.e d = (dt, .... d.), and if d =  for sonm i. define di to be q instead. Then we 
have 
,' dq +v (modl) for mZ +. 
Now fix any choice of m large enough so that we bave 
(5.14) (og ,)   lnaX(«a(),«4{),()) -+1). 
Then by (5.10) we have 
 G [0,1]  x [-(log P)P -k., (log P)P-k] -=  
for this choice of m. Now write  = (d, d2 ..... d, O. 0 ..... 0), where there are R-r 
zeros here, and define ., similarly. Then, setting 
u =  + (0,0 ..... 0, a.{+x) ..... an), 
we have 
a=*d+u (modl), 
q 
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where (dl, d2 ..... d,, q) = 1 and where, bv colnbilfil,g (,5.14) with (5.10), (5.12) and 
(5.13), we have 
l_<q_<(logP)B and u] (logP) . 
Thus recalling dcfinition (&ll), wehave ,  , (qd, P) for out particular 
choice of  and in fact recalling (.12), we also bave   .(P). As in the 
case r = 0, this is a contradiction, whencethe equality (5.2) must in fact hold. This 
completes the proof of Lemma 5.1. 

At this point, we make an observation about the lemma for those familiar with 
earlier argmnents of this type. VVe note that in previous work by the author ([10], 
[11]), the analogue of our Lemma 5.1 was proved with two different methods, for 
two subregions of the region roT(P). If we were to proceed by analog. with earlier 
arguments, we would instead have to treat a region mT, To (P), say, in place of mr(P), 
for positive rem numbers T0 with T0 _< T. The new region would be defined by 
mT,To(P)=mT(P) C {O : ]0] >_ T0}. 
Essentially by combining the arguments used for each region in previous proofs, we 
are able to dispense with the reqlfirement lai > T0. 
Having done most of the work, we tan now give a lemma that essentially savs 
that H IgJ(a" P)I is small for a E m. The idea of using such a lenmla is due 
j=l 
originally to Bentkus and G6tze [3]. 
Lemma 5.5. Deflne the forms Fi(x) as in (1.2) for 1 < i < R and the exponential 
suns gj(. P) for 1 <_ j <_ s as in (4.8), with r i sufjïciently small. Suppose that the 
coefjïciet matrix A associated with t]e sgstem F bas ratk R. Suppose also that 
the irrationalitg condition (iii) of Theorem 1.2 ]olds. Then tbere exists a function 
T(P) that depends only on B.! ad the coefjïcients of the forms F. F2 ..... F, 
that satisfles T( P) >_ 1 and 
(5.15) lira T(P) = 
P-cc 
and such that if we define re(P) as in (4.13) with this choice ofT(P), then one bas 

sup H Igj(,P)l : o(P) . 
om(P) 2î-: 

Pro@ The lemma is very similar to Lemma 6 of [10] and Lelnma 4 of [11], and the 
proof follows iii a silnilar fashion. [] 

We note that this lelmna (and Lemlna 5.1) holds for a.ny positive choice of B, but 
that the function above that is o (P) depends on B. We have stated this lennna 
in a general fashion in the hopes that it nmy be useful for future workers. 
We observe that one could ensure that the function that is o (P») depends only 
on B, r] and 2R - r of the coeflïcients. This follows, with some effort, after finding 
a subset O r C_ {1,2 ..... s} with IJI --= 2R - r such that the conditions Aj(o) ( Q 
for j  J, taken together, imply that c+l ..... cu = 0. This can be proved, 
although out method of proof, at least, is hot straightforward. 
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Iu the remainder of the paper, we fixa flmction T(P) that satisfies the conclu- 
sions of the above lelnma. We note that this is the special function we referred to 
above in section 4, and is used to define the lninor arcs and trivial arcs. 
We observe at this point that we could obtain corresponding results which are 
very similar to Lemmas 5.1 and 5.5 if the expouential sums gj were replaced by 
exponential sums over a complete int.erval. The only major change needed would 
be to use Lemma 2 of [11] in place of out Lelmua 5.4. 

O. THE lk|INOR ARCS 

In this section, our goal is to show that the coutribution from the minor arcs 
to the integral in (4.9) is o (P-nk). We first give a lemma, which is essentiallv a 
restatement of results due to Vaughan [19]. [20], and results due to Wooley [23]. 

Lemma 6.1. S-uppose that k is an nteger with k >_ 2. Define g(c) as in (4.6), 
with q sufficiently small. Then there is an absol-ute positive constant ff' such that 

ff t is a real number satisfying either 
(i) t >_ miu (2 , L'(log/," + h»g log k + 2) + 
\ 

OF 

then one ha 

C'k log log k ' for k > 3. 
log k J - 

(il) t > 4 for k= 2. 

(6.1) ff01 

ig(c01 t , pt-e. 

We observe that one could certainlv improve Oll the lemma iii certain cases, but 
we choose to use only the above bounds for out results. 

Pro@ If the first bouud of coudition (i) holds, then the result is Lenmm 6 of [11], 
which is essentiallv due to Vaughan [19]. [20]. If on the other hand. the second 
bound of condition (i) holds, then we may essentially quote Lenmm 7 of [11], which 
itself follows ahnost immediatelv from work of Wooley [23]. We note that the 3 in 
Lemma 7 of [11] has been replaced bv a 2 here; I aih gratefifl to Scott Parsell for 
showing me the technique one uses to make this inlprovement. 
In the case in «hich (ii) holds, we give a proof for completeness. Define 

e=t-4. 

VCe need only prove that Olle has 
 [g(a)[4+«d a « 
p2+«. 
Clearly. we may assmne that e _< 1 holds. For convenience, we write 
2 

Defiue 

91 = rl ° G [0.1]'lg(cOI > P(log p)-C;}. 
Also, for positive integers m. define 



S_ STEMS OF DIAGONAL DIOPHANTINE INEQUALITIES 2693 

Now for o E -qlm, we apply Lenmm 5.4 with the choice/? = G. Thus, for large 
P and any positive real number , there exist coprime integers a and q with q _> 1. 
and 
<< 2 m(2+6) and ( - î << q-12m(2+6)P-2. 
q 
q 

Thus ,are have 
It follows fol"  < e/2 that 
On the other hand. one has 

(.3) 

But ri0.11 

q 
q2rn2+5) a---1 

« p2+e  2-m(-26) « p2+e. 

I9(a,)lada is less than or equal to the nmnber of sohltions of the equation 

with 1 _< :ri _< P fol" 1 < i < 4. This is bounded bv a constant nlultiple of p2 log P, 
a well-known result, which can be proved bv elementarv means. Thus ff'oto (6.3) 
and the definition of 9l. we have 

[0 Lq(°)[4+¢d  (P(log P)-) p2 log P « P:+¢ (log p)-l, 

by our choice of G. Combining this bound with (6.2) completes the proof of Lenmm 
6.1. [] 

We note that (6.1) is an example of what one might call an "'exact Hua inequal- 
ity". In most work using tlle Hardy-Littlewood method, one uses bounds of the 
type (6.1) where one onh" needs to show that fol" anv « > 0, the left side of (6.1) 
can be bounded bv pt-k+¢. Dispensing with this e is crucial for our work. The use 
of such an inequality stems from the work of Bentkus and GStze [3]. 
For the renlainder of the paper, we now fix a choice of  so that Lemmas 5.5 and 
6.1 hold fol" this choice. Now we turn to what is essentiallv our analogue of Hua's 
inequality. It is very similar to Lemma 8 of [10]. 
Lenmm 6.2. There is an absol,de positive real constater C_ with the followin9 
property: 
Assume that the forms F1. F2 ..... FI are as in Theorern 1.2. with coejïjïciet 
matrix A satisfyig (3.2) and (3.3). Assume lhat f is a positive integer satisfging 
(>_5 for k=2. and 
( 72k l°g l°g k )fork>3. 
g >_ nfin 2  + 1. k(log k + log log k + 2) + log t,- - 
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Define the exponential smns gj(o, t 9) as in (4.8), and deflne the function K as in 
(4.2). Let d(P) be a nonnegative real«,alued.funetion, and let n be any subset of the 
region 

Also, define 

Proof. Observe first that for an3" real number e with 0 < e < 1, one has 
s R 
L II 'gJ(«'P)' 1-I 
j=l z=r+l 
s 
« sup lgj(. P)l 

IK(ai)[da. 

It follows fFOlll trivial estiinates that Olle has 

 R 
j=l i:r+l 

« (h(n" P)P») (P»-gn)- L 

eR R 
H IgJ(«" P)II-e H 
3:1 i=r+l 

We may certailfiy choose a positive real number e so that we have 
(6.4) g(1-e)>4 if k=2. 
Defining C' as iii Lenuna 6.1 and choosing OE2 to be sufficiently large, we lllay ensure 
t hat we have 
( C'kloglogk ) 
.t _> lnin 2 k + 1. k (log k + log log k + 2) + log k + 1 if k _> 3. 
Thus we ma.v choose a positive real nulnber e, slnall enough (in terlns only of k and 
C) so that we have 
( C'k log log k) 
(6.5) g(1 - e) > min 2 . k (log k + log log k + 2) + log k if k > 3. 
In each of the cases, we denote our particular choice of e by u. Now one can join 
the proof of Lemnm 8 of [10] after equation (66), and then follow the remainder of 
that proof with only slight adjustments. The bounds (6.4) and (6.5) are the crucial 
bounds that, we need to apply Lemnm 6.1. V'e omit the details. [] 

Now we can wrap up our work on the minor arcs. We have the following lenmm. 
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Lemma 6.3. Suppose that we are in the sctting of Theorem 1.2 and that the 
coejïficiet matrix A of the system F satisfles the corditions (3.2) and (3.3). Choose 
a function T(P) as ir Lemma (5.53. Define the exponertial sunrs gj(o) as in (4.83, 
with  sufficiently srnall, the region m as in (4.133, ard the function Ix" as in (4.23. 
Then one bas 
s R 
Jm = "3(0)I i=r-t-lH I'(°zi)[d°z =o (tgs-'k). 

Pro@ We siinply apply Lenuna 6.2 with the choices n = m and d(P) = 0. VVe ha.ve 
h(m, P) = o(1) by Lelnma 5.5. Tlms the proof of Lenllna 6.3 is coniplete. [] 

7. THE TRIVIAL ARcs 

Iii this section we show that the contribution frolll the trivial ai'es to the integral 
in (4.9) is o (ps-R), which is now easy to do, having done the necessary work 
above. We have the following lenlnia. 

Lemma 7.1. Suppose that we are ir the setting of Theorem 1.2 ad that the 
coejïficiet matriz A of the system F satisfies (3.2) and (3.33. Choose a functior 
T(P) as in Lemrna (5.5). Defl, e the exponential sums gj() as in (4.8) with q 
suciently small, the region t as in (4.143. ad the functio Iç as in (4.2). Then 
one has 
s R 

Pro@ YVe apply Lenuna 6.2 with tlie choices n = t and d(P) = T(P). 
h(m, P) = O(1) by trivial estiinates. Thus we obtain 
R 
fil IgJ(°t)l H IIx-(°q)'d° «7. (T(P))-IP s-Rk, 
= i=r÷l 
which by (5.15) of Lemma 5.5 is o(PS-Rk). 
complete. 

We have 

Thus the proof of Lemma 7.1 is 

8. THE IklAJOR Aacs 

We now treat the lnajor arcs. Our goal is to show that for large P we have 

s R 
3----1 i=r÷l 

8.1. Approximation on the major arcs. We start our treatment of the major 
arcs by approximating the functions gj(o) by auxiliary functions. We need soine 
notation before we do so. We define Dickman's function p by the conditions 

(8.1) 

p(,) =0 
p(u) ---- 1 
up'(u)-----p(u- 13 
p is continuous 
pis difïerentiable 

for u_<0, 
for0<u< 1, 
for u > 1, 
for u >0, 
for u > 1. 
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Also, for real nulnbers ff, define the functiol 
fo Pk Ç log_x_  
1 oc(l/l)_lp e(:r)dr. 
(8.2) () =  » og P ] 
For real vectors   Nn, define Aj()  in (4.7), and write 
(8.3) wj() = w (Aj()) for 1  j  s. 
Also, for integers q and a with q k 1, define 
= k q 
We now collect sonm results, given by Brfidern and Cook [5], in the following 
1PIllllla. 
Lelnma 8.1. Defit, e 9() as i (4.6) and ([) as i (8.2). Suppose ha¢ a and q 
are iegers with q  1. ad ha  is a real number. The one bas 

and 

oe(fl) « lnin (P, 

Proof. The first result is simply equation (29) of [5]. The second result is essentially 
the third centered equation on page 135 of [5]. [] 
We note that, as remarked by Brfidern and Cook, a and q are hot required to 
be relatively prime. We now state the central lemma of the section, which is ver3" 
similar to Lemma 4.4 of [16]. 
Lemma 8.2. Suppose that we are in the setting of Theorern 1.2 and that the 
coefficient rnatriz A of the system F satisfies (3.2) and (3.3), and that there is a 
real vector z satisfying (3.6). Define the so-called singular series  by 
(3 = 1 r = 0, and 

(8.5) @:Z Z q-S H S (q, AJr)(a) ) for 
q=l a (nod q) j=l 
(a ..... a,.,q)=l 

and the singular integral Z(P) by 

Fiz any positive real number e. Then if P is a sufficiently large positive real number. 
oe bave 

« p-n," ((log p),(+,)- + (logP),(-(e/k)+,)). 
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PTvqf. There are three steps of the proof. One first at)proximates each fimction 
bv terres of the form q-lS (q, A.T)(a))coj(/)on each of the major arcs, then 
gj( o) 
one extends the integration over each nm.jor arc to ail of IR R, and then one extends 
the smn over q to all positive integers q. The argument closely follows the proof 
of Lemma 4.4 of [16]. One major difference involves the use of the approximations 
given in Lemma 8.1. Since we are using exponential smns over smooth numiwrs, we 
need to use these approximations instead of mre standard results for exponential 
sums over complete intervals. Finally, we observe that the only condition we really 
need on f for the purposes of this lenuna is the b,mnd f >_ 2/,: + 1, [] 

Now we turn to consideration of the singular series  and the singulm" integral 
Z(P). In particular, we shall show that we have 9 » 1 and also that xve have 
Z(P) » P-Rk for sufficiently large P. We first treat the singulm" series. 

8.2. The singular series. \'e give some (lefiuitions. Sui»pose that G1, G2 ..... Gr 
are r integral diagonal fOllllS iii s vm'iables with coefficient matrix B, with entries 
dij. hl Section 8.2, we consider integral forms, and of coin'se we aSSllllle that r _> 1 
holds throughout this section. 
\Ve then define the singular series (G} associat.ed with this svstem of r finms 

by 

(8.7) 

where we set 

e=e(.¢)= E q-* H S(q, ab 
q=l a mod q) 
(al ..... a,.,q)=l 

r 
aba)=Edijai for 1 <_j <_s. 
"We return to the notation of the rest of the pat»er for a momcnt. Observe 
that the definition of  given here coincides with the dcfinition (8.5) in the case 
r = R. Moreover, in general, whcn 1 ç r 5 R hokls, the first singular series is 
exactly the latter singular series, where the latter is associated with the first r forms 
F1 ..... F. Note also that the first singular series is independent of the R- r fonns 
Fr+l ..... F. 
Suppose that p is a prime and , is a positive integer. Then we sav that an 
integral vector x = (1, x2 .... , ,) is a solution of rank r (mod p) of the svstem 
of congruences 
(8.8) Gl(x) 
if there is a subset J ç {1, 2 ..... s} with ]JI = r such that one has p  det(Bj) and 
p {  x. Also, for anv prime p and any positive integer n, we define M (p, G) to 
jJ 
be the number of solutions x (mod p) of the system (8.8). 
Now we shall define the concept of a nonnalized systeln of forms.  follow 
Low, Pitman and blff [13] closely, but we need a slightly more general notion. 
 essentially want to define a notion of a system such that a related system, 
which results after setting all but some subset of tf of the variables equal to zero. 
is normalized in the original sense of Low, Pitman and kXlff. 



2698 ERIC FREEMAN 

Suppose that the coetï-icient matrix B contains t disjoint nonsingular r x r subma- 
trices BI, B2,..., Bt. To be clear, t)v this we lnean that there is solne permutation 
of the columns of B so that the first r columns form a nonsingular submatrix, the 
second r cohnnlls form a nonsingular sut)nmtrix, and so on, through the t th set of 
r cohmms. We define 
Now let j = (j, j2 ..... jt) t)e the ordere(l (tr)-tuple su«h that the particular matrix 
[B1B2... Bt] is the sut)matrix of B consisting of the columns of B indexed in order 
t)v j,j .... ,j; that is, we define j so that for 1  v  t and 1  b  r. the 
-th cohmm 
]t,(r-l)+h 
dcpcnds on j. Also set 
. = {j. j,..., j,}. 
Suppose that p is a prime dividing A. Here we define, following [13] closely, 
a p-operation on the fornls G1 ..... G r aS a transformation that produces integral 
forns H .... , H, and has the fidlowing steps: 
(i) Pre-multiply B by an integral unimodular matrix U with entries in the set 
{0. ..... 
(il) Next, multiply at most tf" - t" of the cohuims of U.I by pk and multiply 
anv of the colunms of U{1 ..... s}kJ by pk; 
(iii) Then divide g of the rows by p. where we have 1 N g  r. 
As discussed in [13], step (i) corresponds to adding linear combinations of some 
of the forms to one or more of the other forms. Step (ii), on the other hand. 
corresponds to writing xj = pgj in each cohuun j that one multiplies by pk, and 
then trying to solve the new inequalities in the variables yj. Step (iii) corresponds 
to dividing g of the r equations by p. One can check.  in [13]. that a p-operation 
is possible for all primes p that divide 
Note that for the resulting system H. we bave 
for sonle ilteger m.  say that such a p-operation is permissible if one h 
m<0. 
Observe that. upon perfornling permissible p-operations for anv of the primes 
p dividing A(G), we can find a system H that can be obtained from the original 
system G via a finite sequence of permissible p-operations and such that A(H is 
minimal. If G is a system of r integral forms as above, such that A(G) cannot be 
reduced by any permissible operations, then we say that G is a (j. t)-norlnalized 
system. Finally, if G is a system of r integral forms in exactly tr variables, then 
we simply say that G is a normalized system.  note that. in this case, our 
definition clearly agrees with the definition given for a normalized system in [13]. 
e make one other observation. Suppose that.  above, the coecient matrix 
B of a system G, in s variables with coeNcients dij, contains t disjoint nonsingular 
r x r sublnatrices B, B,..., Bt, and define j and J as above. Then. we define the 
system G* in tf variables, by defining, for y  Z t and for 1 < i < r. the forms 
(8.9) 
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Note that this is simply the system obtained bv setting ail variables with indices 
j  J equal to 0, and subsequently reordering the variables. Observe that the 
coeffiçient matrix B*, sa)-, of the system G* has the form 
B*=[ B, B2 Bt ]. 
Suppose nov« that a system H can be obtained rioto the system G airer a finite 
sequence of permissible p-operations, and suppose that H is (j. t)-normalized. Then 
consider the system H*, defined as in (8.9). We can sec that the saine p-operations 
(restricted to the columns j E J) allov« one to obtain H' from the system G*; airer 
all, the variables of G* are a subset of those that appear in G. Each operation is 
certainly still permissible, since the definition of A involves only the coluinns j E J. 
If one could reduce A eH') via a permissible p-operation, then we could simply 
extend step (ii) and nmltiply all of the cohmms of UB{ ..... s}\J by p#. This would 
give a permissible p-operation for the system H, which contradicts our assmnption 
that H is (j. t)-normalized. So if H is (j. t)-normalized, tllen it follows that H' is 
normalized, and moreover, if H results from G airer a finite sequcn('e of permissil»le 
p-operations, then H' results from G" ri'oto the saine sequence of p-operations. 
We can now sIate the following lemma, which is a step towards bomding the 
singular series below. We do slightly more than we need to, in the hope that it vill 
be usefifl for fiture workers. For this reason, we state I he lemma in a self-contained 
lllallller. 

Lemma 8.3. Svppo.e that r, k and s are positive integers with k >_ 2. and suppo.e 
for l < i < r that 
D,(x) : dilfl" Î ÷ d,22"  +... + dis.r  
fs an integral diagonal form of degree k. Suppose that s satsfles s k tr. where t fs 
an iteger satisfying 
t>2k+l. 
Suppose that the coecient matrix C of the system D contais t disjoint nosin9ular 
r  r submatrices C1, C2 ..... Ct. and deflne 
t 
A(D) =  
Also, define the sig.ular series (D) as m (8.7), Suppose that the followmg prop- 
erty, which we denote by Pet, k, r), holds: 
Given ang system of r itegral diagoml forons G1. G2 ..... Gr of degree k in 
tr variables, with coeciet atrix B which consists of t disjoint osingular 
r  r submatrices and such that the system G is ormalized. then for every 
prime p and every positive integer n. there is a solution x of tank r (mod p) 
of the system of congrueces 
Gl(x) G2(x) --- oeGr(x) 0 (nlodpn). 
Then the series (D) converges absolutely and one bas 
(D) >>D 1. 
If one also bas 
t>kr+k+l, 



Proof. We first give some more notation. For anv prime p, define 3' = "(k.p) bv 
«hoosing - to satisfy p'-II/,', and settiug 

{ 1 if T=0 
-) = "r+l if 7->0andp>2 
T+2 if "r>0andp=2. 

Also. for anv l)rime p. we definc 
(8-J 1)) \D(P) ---- 

.'il (p'. D) 
__,oe pn(s-r) 
As iii Q h«l|)t( r 5 of [61, and ll,'qing al,,qo Lenllna 2.1() of [15] OllC lllV see that this limit 
exists, that (D) converges al)solutely, and that @(D) is equal to an absolutely 
('(llV('l'gellt l)r()(luct, that is. we have 
(.) (D) = H 
P 
(XXb noie that it is in his argument that one uscs the condition t k 2k + 1. and 
that he rate at which the product converges deI)Cnds on D.) 
Now dcfine D as in (8.9), and dcfine j and J as in the discussion above. 
may find a (j. t)-normalized svstem G. which can be obtained from D alier a finite 
sequence of I)ermissible p-operations. As we bave noted above, G* is then a nor- 
malized system, which ont obtains from D* aftcr (essentially) the saine permissible 
p-operations. 
Since proper/y P(t. k. r) holds, there is, for ail p and -n, a solution w of rank r 
(mod p) of the svstem of congruences G*(w)  0 (mod p). Bv the way G* w 
dcfiued, one can s that if y = (y, ye ..... y) is defined bv 
{ wj if j  J 
.v = o if j C j. 
/hen we have that y is a solution of tank r (mod p) of the svstem of congruences 
G(y) oe 0 (mod p). In t)articular, this holds for n = 7- XX mav thus apt)ly Lelnma 
6 of [131 to the system G. whence we have 
(8.12) 5I (p,G) k p(»-)(-) tbr n > 7- 
Froln this fact we will deduce a lower bound for 5I (p, D). 
To this end. suppose for some positive integer r tha.t y  Z  is a solution of the 
COllgrHellCeS 
Gi(y) Ç2(y)  ""Ç,-(Y)  0 (mod pn). 
Recall that the svstem G resulted from D after a finite sequence of permissible q- 
operations. Let H be a svstem such that G arises from H after a single permissible 
q-operation. Let I be the subset of {1, 2 ..... s} consisting of the cohmms affected 
bv step (ii) of this q-ot)eration, that is, let I consist of the indices such that the 
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corresponding cohmms in step (ii) are nmltilflied by q. 

x = (Xl,X2,...,a's) by setting 

We show that one has 

{ qyj if j C I 
a if jI. 

2701 

Then define the vector 

(8.13) H, (x) = Ha(x) =.-- --- Hr(x) _= 0 (mod 
To see this, let H (0 and H (i) i,e the syst.ems that result a.fter steI)S (i) and (ii) 
of the q-operation, respectively. We certainly have H¢ii)(y) = 0 (mod t/): indeed. 
sonle of these forlns are congruelt to 0 (lnod pq). Then observe that we bave 
H(ii)(y) = H(i)(x), whence H(i)(x) 0 (lnod p) holds. Since the lnatrix U is 
unilnodular, so that in particular its deterlninant is hot divisible by p. one has that 
(8.13) holds. Thus any solution y of G(y)  0 (mod p) gives lise to a solution x 
of H(x)  0 (mod p). If q  p holds, we therefore have 
M (p",H) k Al (I,',G). 
If q = p holds, we might have some reduction in the nulnber of solutions, because 
lnultiplicatiol by p in /pn has kerlml of size p. but we certainlv bave 
M (p", H ) 
So, by repeating this analvsis for each permissible q-operation, one can see that 
if G is a (j. t)-normalized system arising from the svstem F after a finite sequen«e 
of permissible q-operations, then one has 
M(p,D) OE if p'[[(D). 
pro(s-r) 
Now the linfit k(P) exists; this follows in nmch the saine wav as the corresponding 
fac for D. It follows that we bave 
(8,1) k(P)  p_) if pH(D). 
It follows from (8.12) and the definition of k(P) tiret for ail primes p we bave 
(8.15) kD(P)  p-(v+ord(aD)))(-). 
Since the product  kD(P) is absolutelv convergent, there is a constant c(D), 
which mav depend on D, such that we have 
1 
Thus, using also (8.15). we have 
1 1 
pNc(D) pNc(D) 
1 
 (A(D)) -  p-(- »D1. 
pSt(D) 
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Now suppose that we have t >_ kr + k + 1. One can prove as iii chapter 5 of [6] 
that for primes p, one has 

(8.16) 
n=l 

where for positive integers , we define 

s (p') = 

Z H S(p''Mj(a))" 
a (mod p') j=l 
(al ,...,a,p)=l 

Now suppose that p{ A(D). Sut)pose for 1 _ v  t that Cv consists of the 
cohmms j,,jv .... ,j, in that order. Then for a satisfying (al, .... a,.,p) = 1, 
there nmst exist sonle j  {j, ..... j. such that p  ]iii(a). since we have p { 
det(C). Thus for any prime p with p  A(D), and any positive integer , by the 
standard estimatc S (p', a) « p,(1-(1/¢)), which holds for (,p) = 1, we have 

S (p') << p- 

Z pnS-(nt/¢) « pn(r-(tl¢)). 
a (mod 
(al ..... 

Combining this last bound with (8.16) and using t >_ kr + k + 1 yields 

\D(P)- i EE Zp nr-(t/k)) EEp -1-(1/k) for p{A(D). 
n=l 

So there is a constant C that depends only on k, r and t such that one has 

(8.17) I\D(P) - 11 < CP --(/) for p{A(D). 

Now, because Z Cp-I-(1/k) converges, there is a constant C that depends onlv 
p 
on k, r and t such that one has 1 - C -1-(1/t¢) > 0 for p > C and 

1 
H (1- Cp -1-(1/k)) _ . 
p> 

Now for all p we have \D(P) > 0 ff'oto (8.15); so by (8.17) we have 

plA(D) p'A(D) 

plA(D) pA(D) 
1 
 I-[ (1 I-[ D()" 
p[A(D) 
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It follows from (8.15) that we have 
1 
p p<(, 
p/XID) 

H 

p--('+ord,,( A(D) ) )(s--r) 

(A(D))T-* H P-«-) 
plA(D} 

(A(D)) '-* (A(D))20 "-) " 

This completes the proof of Lemma 8.3. 

Now we give another lemma that builds on the above lemma and completes the 
treatment of the singular series for the cases k _> 3. As in the case of Lemnm 8.3. we 
do slightly more than what we will need, and we stale the lemlna in a self-contained 
fashion. 

Lemma 8.4. Suppose that r, k, and s are positive integers, ad suppose for 1 <_ 
i < r that 
Fi(x) = Aixî + Ai2x2 +... + Aixs 
is an integral diagonal form of degree k. Suppose that the coefficient matrix 4 of 
the syste F contains g diooint nonsingular r x r submatrices .4 ) A () AW) 
where g is a positive iv teger satisfgig 
fk2k+l. 
«  = H ]dot (A}:')]. Define  = (F)as in (8.7). Finally, suppose that 
one of the two followin 9 statements holds. 
(i) k is odd, and k  3 holds, and one bas g  kmo, where mo is the least 
positive integer m such that one bas 
(ii) k 2 3 holds, and one bas 
g > k [48k = log 3rk =] . 
Then oue has 
O(F) » 1. 

Moreover. if 

g>kr+k+l 
holds, then there exists a positive real constant c(k, r, s) that depends only 
o k, r and s such that one bas 
rg(F) _> c(k, r, s)A -. 
Moreover, we note that if k is odd with k >_ 3. then there exists a absolute positive 
real constant C such that condition (i) holds if one has 
rk log 2/," 
(' > + Ck log(r log 2k). 
- log 2 
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Proo.f. The last statement of the lemma can be checked with a straightforward 
computation. 
Thus to prove the lemnm, we need only check that the condition P(& k.r) of 
Lemma 8.3 holds for out choices of (. So suppose that G is a normalized system in 
l'r variables. For the case in whi«h condition (i) holds, we may apply Theorem l(ii) 
of [13] to see that for any positive integel" 71, the svstem G(x) = 0 (mod p') has a 
solution x of tank r (mod p). On the other hand. for the case in which condition 
(ii) holds, we may apply Theoreln 3(i) of [13]. 
Tlms in either case, the condition P(& k, r) of Lemma 8.3 holds, v«hence Lelnma 
8.-1 follows. 

Now wc give a lclmna to treat the singular series in the case k = 2. \Ve observe 
that one could surely obtain a rcsult that is better for large r, but we choose hot 
to pursue this hcre. 
Lemlna 8.5. Suppose lhat r and  are positi,e bdeger.s, and suppose for 1 < i < r 
that 
Fi(x) = )il.Fî -- )i2J'72 --...-- 
i.s an itegral diagonal quadratic form. Suppose that the coeJficient matrix A. defined 
as in (1.3). contams g disjoint 71oT,s'ilgular r × r submatrices, where  is an integer 
sati.sfying 
t' OE lllill (-11.2 + 41" + l. 38-1 log 161" + 5). 
Define t = t(F) as i'n (8.7). Then one bas 

5(F) »F 1. 

Pro@ If the first bound for  holds, then it follows that any nontrivial complex 
liniar combilation of the forlns F ..... F bas rank at least 4r 2 + 4r + 1. Bv the 
theorem in [18], the singular series t is positive and depends onlv on the forms 
F1, F2,.... F,-. One can readilv check tllat the singular series t is defined in [18] 
in the saine ruminer as we have defined it. 
Sul)l)ose instead that the second bound for  holds. \Ve give a sketch of the 
proof in this case. We first seek an analogue of Lemma 12 of [13] for the case 
k = 2, with m_>[1921og16r+2]. For primespwithp<_81.2 , say. it is easv to 
check that Lenmm. 5 of [7] provides an analogue of the desired type. To obtain 
an appropriate analogue of Lelnma 12 of [13] for primes p > 8r 2, one applies an 
adaptation of Theorem 2 of [14], «ith, say, c = 4: one can check that if one assumes 
that the matrix of coefficients contains c + 1 nonsingular i" × r submatrices, rather 
than assuming that the matrix is highly nonsingular, then the result still holds. 
(This can be seen by noting that the inequality q(B) > ci, which would still hold, 
is the key condition needed on page 339 of [14].) 
In either case, we bave an analogue of Lemma 12 of [13]. and thus ce can show 
that condition P(tç, k, r) holds, a.s in the proof of Theorem l(ii) of [13]. [] 

Michael Knapp and Professor Wooley provided me with a proof of a result closel3 
related to the second part of thc above lemlna, for which I alll grateflfl. 
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8.3. The singular integral. F{ecall that in (8.6) we defined the singular integral 
Z(P) 
-(P) = . Çj=IIlt'UJ(fl) ) (i=rl[Ç(Çi)) d" 
Our goal in this section is to demonstrate thN for large positive P we have the 
bound 
(8.18) g(P) »F P-- 
Instead of using the traditional approach which uses Fourier's Integral Thcorem, 
we use a method given by Schmidt [18]. Below we [ollow parts of [18] very closely. 
Much as in [18], for anv positive rem mmbcr T and anv rem numbers a and , 
we define 
2 
lçT(O)=lç(aY 1)= ( sill ( -1) ) 

l.lld 
(8.19) 

(8.o) 
We now define 

= j" T(1-Tlfll) for Ifil-<T-1 
¢'T ( fl ) 
0 for [fl[ > T -1 
From (4.3), one may readily deduce the %llowing identity, which holds for all rem 
nmnlmrs , namely, 
@T(fl) = [" e(al3)Kr(a)d«. 

By (4.4) and a similar bound for tçT(a), the integral converges absolutely for each 
choice of P and T. 
We shall see that for fixed P, we bave liln ZT(P) = Z(P), and we will also show 
that for large T. we have ZT(P) »F p-Rk. These two facts together establish the 
bound (8.18). To prove the first fact, we give a bound for the difference ZT(P) -- 
Z(P). 
Lemma 8.6. Sppose that T ad P are positive real numbers with 
P >_ 1. Sppose that R. r. I; and s are bte9ers with R >_ 1.0 < r < R ad k >_ 2. 
S'uppose for 1 < i < R that 
Fi(X) = ,il,Z'Î ÷ ,i22F22 ÷... ÷ 
is a real diagonal form of degree k and that for 1 < i < r, the form Fi is itegral. 
Asstrne that one bas IIF]] >_ 1. &tppose also that the coeJficient matrix A of the 
systern F is as in, (3.2) ad satisfies (3.3). where one bas 
g,>k+l. 
Define Z(P) and ZT(P) as in (8.6) a«d (8.21), respectivel. Then ote bas 
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We note that the implicit coTstont in Vinogradov's notation bere depends af most 
on H. r. k and s and, in particular, does hot depend on the coeciets of F. 
Pro@ Observe first that, in the case r = 0, we havc Z(P) = ZT(P). So we can 
assume that we have r  1. It follows from the definitions (8.6) and (8.21) that we 
have 
From the pemfltimate centered equa.tion on page 305 of [18], one has 
1--HIçT()«T -e max [e«T-el] e for 1[ <T. 
i=1 
F 
and for ]] OE T, one clearly has  lç() (( 1. çoml,ining these 1)ounds with the 
estimatc for w() given in Lemma 8.1, and thc 1)(rond (4.4) [or Iç(o). one h 

Consider the first integral on the right-hand side of (8.22). By H61der's inequality, 
one has 

(8.23) 

j=l 

« H ,<T ]/]2 H lllill(P']AJ(t)l-l'k) gdt 
v=l j=(v--1)lR + l 

Fol" a fixed choice of v with 1 < v _< & one Inakes the change of variable "7 = 
(71 .... ,TR) = U,,(/3) given by 7j = A(,,-UR+j(/3) for 1 _< j _< R. and obtains 

j=(v--1)R÷l 

R 
'u-l(') 2 H 111i11 (P, Ij[-1/k) ' 
3=1 

If "7 = U. (/3), then one has 
l'ri = IUv()l < IIFll- 
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and by Cramer's rule and Hadamard's inequality, one has 

(8.24) 
It follow8 that we lmve 

(8.25) 

AI' 

But one has 
SI-ri< RIIF[[T 

d7 

whence we have 

R 
j=l 

since we have 7 > k + 1 and IIFIIT _> 1. 
Thus, by (8.22), (8.23) and (8.25), we have 
2c(P)- 2eT(P) << IIFII2ST-11p s-sk 1-I A73/g 
+ Ps-Rgf, O H rein (P, IAj(fl)] -llk) dfl. 
I>_Tj=t 
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By Hiilder's inequality and a change of variable as al»ove, we have 
:Z(P)- :Zr(P) << IIFll2"T-1/"P s-' H AÇa/e 
,(/,.  
r=l I'l--cFa'T j=l 

where CF,v is a positive constant which by (8.24) we lnav defilm by 
«e,v = RR/=IIFI]R_ l- 

d7 

since we have t e >_ k + 1. Combining the last txvo bounds completes the proof of 
Lclllllla 8.6. [] 
Now we prove a lemlna which states that for T a.nd P sufficientlv large, the 
quantity ZT(P) is bounded below. 
Lemma 8.7. Suppose that we are ir tbe selti9 of Tbeorem 1.2 and that tbe 
coefficient rnatriz A of the sgstern F satisfies (3.2) and (3.3). and that there is a 
real vector z satisfying (3.6). Suppose also that one has IIF[[ > 1. Define At as in 
(3.3). Suppose that T aM P are real vurnbers satisfying T > 1 and 
(12 )1/ 
(8.26) P > -'n/?2R]lF[[2/lnax(A-2,1 ) 
where 6 is as in (3.6). Define 2-r(P) as in (8.21). Then there is a constant ci = 
q(F,k, R.r,s, 5. z) that does hot &pend on T or P such that one bas 
çT(P) OE Cl Ps-Rk. 

Pro@ Recalling the definition (8.3) of wj(13), we can write the absolutelv conver- 

,=i)1 ( og«  
r 
t=l i=r+I 
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Using the identities (8.20) and (4.3), raid recalling the definition (3.4) of Li(x), 
we can rewrite ZT(P) as 

1 )l/k-1 IïI p \klogP 
- ,pkls(.rl-.-Xs )I*',T(Li(x)) 1Ç[ g'(Li(x))dx. 
j=l t=l i=r+l 

ZT(P) is certainly larger than the corresponding integral over the slnaller region 
__.pk , noting froln Lelnlna 12.1(i) of [21], alM (8.1), that the functiol p 
is always nonnegative. Fol P satisfying (8.26), one certain|v has P > (5/2) -l/k, 
6pk 
-- < xj < pk, it follows ff'oto (8.1) that one has 
whence for xj satisfying 2 - - 

Ç log Xj " 
 ] = - 

So we have 
z.(P) > ,;. .] 
Now define T4p.T to be the region 

min\l'A//'() fol'l</<," and 
37" 

 ( ) } 
IL(x)l < mil, 1.A'l/ fort + 1 <z < R 
It follows froln the definitions (8.19) and (4.3) of vr(a) and b(a), respcctiveh, that 
we have 

(8.2/) 

T(P) 2> ps-skTr]t s (TPp.T), 

where Ps denotes s-dilnensiolml Euclidean lneasure. 
We now lnake the linear clmnge of variable w = V(x) given bv 
Lj(x) for 1 _< j <_ R 
w j= .r for R+l <_j <_s. 
Since A1 is nonzero, we can see that we have 

(8.28) /£s (7-P,T) >F //s 
where we define $.7 to be the region V(74p.T). Note that $ T is the set of w 

[î ] 
such that there exists an x 6 , Pk 

with w = l(x) and such that one has 

lnin(1./N}/R) lnin(1,zll/R) 
I**'1 < 3T fol. 1 < i < r and Iwl < 3 

for r+l < < R. 

Now we give a lenuna, which is essentially due to Nadesalingaln and Pitlnan. 
(SeP [16], Lellllll& 5.2.) 
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Lemma 8.8. Let R and s be positive integers satisfying s > R. Let 
A = 
be a real R x s matrix. For 1 
Let A dcnotc the absolu.te value of the detetinant of the lefl-hand R x R subatrix 
of A. Suppose that we bave A > O. Additionallg. suppose that Q is a real number 
satisqing 
(8.29) Q 
Suppoe also that w,..., wn are çeal number satisfying 
]wl< for l < i < R. 
- 3 
Let 5o = So(W,..., wn) be th« set of all real vectors (yn+ ..... y)  [-Q. Q]Æ-n 
for which there exist real muber y .... , y 
i<R. 
Then So bas (s - R)-diut«#sional uteasure satisfging 

where the implicit constant 
entries of .4. 

l's-n(So) >>A 
l/ïnogradov's notation depends on s and R and the 

Proof. We apply the lemma of Nadesalingam and Pitman to the R linear forms 
Ml(y) .... ,]ln(Y) defined by ]l,(y)= Al/nLi(y) for 1 <_ i _< R, in order to 
relax the requirement A1 _> 1 of their lemnm. We note that there is a slight 
difference between the definition of [[All that ve use and the definition they use, 
which accounts for the change in the condition (8.29). Here xve have also ilnplicitly 
used the last equation on page 704 of [15] to show that the terln H(L) in the lelnnm 
of Nadesalingain and Pitman is positive. [] 

Now we return to the proof of Lemma 8.7 and apply Lemma 8.8. By (8.26) and 
the assumption T _> 1, we mav apply the lelnlna, with the choice Q = (SP )/2. 

for any w = ('1, t/-'2 ..... wR) with 
(8.30) 
lnill (1,/NI/R ) 
3T 

We obtain 

forl<i<r, and I wil< 

rein (1" Al/n) for 

3 

r+l<i<R. 

p,Æ-n(So(W, .... wh)) »  pa(,-n). 

Now, for anv choice of (Yn+, Yn+_ .... , Ys)  0(iI'l, .-- , wR), there exist real llUm- 
bers Yl,...,Yn e [-Q, Q] with Li(y) =wi for 1 < i < R. Defining z as iii (3.6), ,ve 
h ave 
Li(Pez+y) =wi for 1 <i</ï'. 
By (3.6) and our choice of Q. we also have 

P z + y  [  P , I + a P ] C [  P P  ] 
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Recalling the defilfiti(m of SP, T, we sec that we have 
lts (SP, T) >>F, T-"P 
Cmbining with (8.27) and (8.28), we see Ihat Ihere is a positive rem constant 
cl = ci(F, k, R, r, s, 5, z) such that one has 
(8.31) :T(P) >_ Cl Ps-Rk. 
This completes the proof of Lelmna 8.7. [] 
Combining Lemmas 8.6 and 8.7 yields the following lower bound for 2-(P). 
Lemma 8.9. Suppose that we are in the setti9 of Theorern 1. ad that the 
coefficient ruatriz A of the systeru F satisfies (3.2) and (3.3), and that there is a 
real vector z satisfyin9 (3.6). Assume also that we bave IIFII > 1. Define if(P) as 
in (8.6). Then there is a constant ca = ca(F, h, R, r, s, 6, z) such that for P > ca, 
ont? bas 
Z(P) » ps-Rk. 
Here the iruplicit constater m l ïnogradov's ot«tion may depend on F, k,s. 
and the special real vector z. but t does hot depend on P. 
8.4. Completion of the treatment of the major arcs. We wrap up our work 
on the major arcs with the following lemma. 
Lemma 8.10. Suppose that we are in the setting of Theorem 1. and that the 
coefficient matrix A of the system F satisfies (3.2) and (3.3), and that there s a 
real v«ctor z satsfymg (3.6). Assume also that we bave IIFI[ _> 1. Then there are 
constants c4 and c, whzch may depend on F, k, s, R, r,  and z, but which do hot 
depend on P, such that for real numbers P satisfymg P >_ ca, one bas 
s 
f. 1-[ -(-/ 1-[ K(,)d. _> cP-. 
j=l i=r+l 

Pro@ Choose e = 1/(2k) and apply Lemma 8.2. Since we have g _> 2h + 1 and by 
the definition (4.10) of B, we obtain 
.« R 
(8.32) /.M HgJ(a) H Iç(oi)da-Z(P) « PS-nk(logP)-/(st¢(l+l)) 
j=l i=r+l 
Since condition (iv) of Theorem 1.2 holds, one has  » 1. By Lemma 8.9. there 
are constants c3 and c that do hot depend on P such that one has 
(8.33) Z(P) >_ cP -Rk for P _> c3. 
Lemma 8.10 follows from (8.32) and (8.33) and t.he bound 1,9 » 1. [] 

9. COMPLETION OF THE PROOF OF THEOREM 1.2 

In this section, we gather together all of our results in order to complete the 
proof of Theorem 1.2. 
We recall that we demonstrated in Section 3 that we may assume that we have 
e = 1, that we have ]IFII _> 1, that the coefficient matrix A of the system F satisfies 
(3.2) and (3.3), and that there is a real vector z satisfying (3.6). 
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\Ve first observe how one proves the last sentence of Theorem 1.2. namely that 
and we define m0 (,'.'.) as in (1.4) a,,d assume that we bave 
if 
have 
we 
1. 
HI 0 
8.4, noting that we «ertainly have {  2k + 1 for a suciently large «hoi«e of the 
constant C. For k = 2, we may apply Lemma 8.5. Thus we have  >>F 1. 
Now we turn to the central result of Theorem 1.2. ecall from (4.9) that we 
have 
s R 
(p)  ,.  g)(o)  Iç()da, 
(9.1) 
' J--I i=r+l 
wllere (P) was dcfincd to bc thc numbcr of solutions of the svstem (4.1) with 
.r  M (P, P) for 1  j  s. 
We first choose a flmcti(m T(P) as in Lcmma 5.5. Xç can now treat the minor 
arcs and trivial arcs. Bv Lemmas 6.3 and 7.1, one obtains 
s R 
X now «onsider the major arcs. Bv Lemma 8.10, we have 
" j=l i=r+l 
for P  ca, where ca and cs are constants that do hot depend on P. Together with 
(9.2), it follows for su«iently large P that one has 
 C5 ps--Rk 
By (9.1), for suciently large P, we have 
This establishes Theorem 1.2. 
As a final observation, we note that we llave obtained a lower bound of the 
expe«ted order of nmgnitude for the number of solutions of out svstem in a box 
of size P, for ail su«ient.ly large positive P. Recall that we assumed that we 
bave e = 1, that we bave I[Fll  1, that the coe«ient matrix A of the system F 
satisfies (3.2) and (3.3), md that tllere is a real vector z satisfying (3.6). Using 
standard t.e«hlfiques, one can check that under the conditions of eitller Theorem 1.1 
or Theorenl 1.2, without mly of these simplifying assumptions, the sanie lower bound 
holds for suciently large P. X note that in this case, P nmst be suciently large 
also in erms of e, and the implicit constant in the lower bound for (P) depends 
Oll . 
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ON THE CANONICAL RINGS OF COVERS 
OF SURFACES OF MINIMAL DEGREE 

FRANCISCO JAVIER GALLEGO AND BANGERE P. PURNAPRAJNA 

ABSTRACT. In one of the main resuIts of this paper, we find the degrees of the 
generators of the canonical ring of a regular algebraic surface X of general type 
defined over a field of characteristic 0, under the hypothesis that the canonical 
divisor of X determines a morphism ç from X to a surface of minimal degree 
}'. As a corollary of our results and results of Ciliberto and Green, we obtain a 
necessary and suflïicient condition for the canonical ring of X to be generated in 
degree less than or equal to 2. We construct new examples of surfaces satisfying 
the hypothesis of our theorem and prove results which show that many a priori 
plausible examples cannot exist. Our inethods are to exploit the Oy-algebra 
structure on ç,Ox. These methods have other applications, including those on 
Calabi-Yau threefolds. We prove new results on homogeneous rings associated 
to a polarized Calabi-Yau threefold and also prove some existence theorems for 
Clabi-Yau covers of threefolds of minimal degree. These bave consequences 
towards constructing new examples of Calabi-Yau threefolds. 

INTRODUCTION 

The canonical models of surfaces of general type have attracted the attention 
of lnanv geolneters. The questions on projective normality and ring generators 
of the canonical ring are of particular interest. Kodaira [Kod] first proved that 
[K.ç ' ] embeds a lnilfilnal surface of general type X as a projectivdy norlnal varietv 
for ail m _> 8. This was later improved by Bombieri [Bo], who proved the saine 
result if m _> 6, and by Ciliberto [Ci], who lowered the bound to m _> 5. We 
proved in [GP1] more general results on projective normality and higher syzygies 
for adjunction bundles for an algebraic surface. As a corollary of these results 
we recovered and ilnproved the results of Bombieri and Ciliberto on projective 
normality, and extended them to higher syzygies. We also recovered and extended 
the results of Reid [RI on the ring generators of the canonical ring of a surface of 
general type. 
An important class of minimal surfaces of general type comprises hose whose 
canonical divisor is base-point-free. Surfaces with base-point-free canonical divisor 
fall naturally into two categories corresponding to the division of curves of genus 
greater than one into hyperelliptic and non-hyperelliptic: hose whose canonical 
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lnorphisln maps onto a sm'fa.ce of lninilnal degree a.nd those whose canonica] mor- 
phism does hot nlap to such a surface. By a surface of minilnal degree we mean a 
nondegeuerate surface iu projective space whose degree is equal to its codimension 
pins 1. The surfaces of minimal degree are classically knowu: they are (linear) P2. 
the Verouese surface in pa, and smooth rational scrolls or coues over a rational nor- 
mal curve (see [EH]). Note that, even though we are drawing an analogy between 
hyperelliptic curves and surfaces of general type whose canonical morphisin inaps 
outo a surface of miuima.1 degree, the theory is nmch harder for surfaces, because 
higher degree covers are involved. Examples of such covers are shown in Section 3. 
In this paper we deal with surfaces of general type whose canonical morphism 
maps onto a surface of miuimal degree. The surfaces whose canonical morphism 
does hot mal» onto a surfitce of minimal degree have been studied bv Ciliberto aud 
Green (sec [('i] and [G]). (',reen and Çiliberto proved the following beautiflfl result 
regarding the geuerators of the cauouical ring: 
Let .ç bc a r«gtlar su':face of gcnc'ral t.pc witt basc-point-free canonical divisor. 
.4ssune that the caonical ltO't]istt ç satisfies the following coditions: 
(1) ç does ot ,nap X generic«dlg 2 : 1 onto the projecti,e plane: 
() ç(A-) is ,wt a surface of mi,smal degr«e (other tfian liwar P). 
Thon thc «am,i«al rig of X is gcnerated in dcgre Icss than or cqual to 2. 
The smfaces of general type X whose cauonical morphism ç maps X outo a 
others, where they play a central role in the classification of surfaces of general type 
with small cî aud in qucstious about degenerations and the moduli of surfaces of 
general type. The studv of these sm'faces h a direct bearing on the studv of linear 
series on threefotds such as Calabi-h%u threefolds, as the resnlts in [OP], and the 
authors" results in [GP2] and Sectiou 4 of this article show. 
The stmty of the ca.nouical rings of these surfaces is carried out in Section 2. 
determine the precise degrees of the generators of its canouical ring (see Theorem 
2.1). The answer depends on the degree of ç and the degree of Y. As a corollary 
of out result and the result of Çiliberto and Green (see Corollary 2.8). we fiud that 
conditious (1) aud (2) above characterize the regular surfaces of geueral type with 
base-poiut-fi'ee cauonical lmndle whose cauonical ring is generated iu degree less 
than or equal to 2. This result is surprisiug, because it contrasts with the situation 
for higher-dimeusional varieties, which, as ve show in [GP3], differs Dom the sit- 
uatiou for smfaces, ludeed, ve show lu [GP3] that there is uo higher-dimensional 
aualogue of Corollary 2.5, aud therefore there is no converse of Green's result (cf. 
[G], Theorem 3.9.3) for higher dimeusioual varieties of geueral type. 
Iu Section 2 we explain how to use the Oy-algebra structme ou ç.Ox to fiud 
the multiplicative structure of the canonical ring of X. Even though we reduce 
the problem Dom a complicated variety to a simpler variety.  a surface Y of 
minimal degree is, there are ceïtain diculties that arise in the process. The 
proof of Theorem 2.1 involves the study of multiplication of global sections of line 
bundles on a smface X of general type. To do so we reduce the problem to 
and this amonnts to stndying maps of muhiplication of global sections of vector 
bundles instead of line bnndles. This is the first dicnlty. Moreover. the relation 
between the mnltiplication maps of global sectious of line buudles on X aud the 
multiplication maps of global sections of vector bundles on  is goverued bv the 
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O--algebra structure on ,Ox. However, we prove our results for a large class of 
surfaces X. Se their canonical lnorphisln g) niight, and a«tually does, correspond te 
niany, quite diverse Oy-algebra structures. This is in sharp «ontrast te hyperelliptic 
«urves, where canoni«al nmri)hisms correspond te degree 2 algebra structures, which 
are ail quite similar aud very easv te des«ribe. The algebra structures arising 
frein canonical niorphisms of surfaces are niuch lnore «ompli«ated and liarder te 
determine. This is the second difficulty one encounters, a diffi«ulty that we are able 
te overcome in the «ontext of this Imper. 
In Section 3 we «onstruct new exaniples of surfaces of general type mapping te 
a surface of niinimal degree and also re«all seine known eues. It is interesting te 
know in general what positive integers o««ur as degrees of the canonical morphisni 
if the image is a surface of minimal degree. An answer te this question is helpflfl 
in finding new exmnples. Having this I)hilosoI)hy in nfind, we prove resuhs which 
show that seine natural wavs te «onstruct examples de net work. For instance, il, 
follows ff'oto the results in Section 3 that odd degree «overs of smooth scrolls or 
«vcli« covers of degree bigger than 3 of surfaces of mininml degree, indu«ed bv the 
canoni«al morlhisni, de net exist 
The results on surfaces of general type mentioned above have ramifi«ations for 
Calabi-Ymt threefolds. In Sectkm 4 of this article we apply these results te obtain 
new results for a polarized Calabi-Yau threetbld (X, B) with B a base-point-free 
and ample diviser. Aniong other things, we find out the degrees of the generators of 
the homogeneous ring asso«iated te B, and we give a chara«terizatiou of polarized 
Calabi-Yau threefolds (X. B) whose asso«iated holnogeneous riug is generated in 
degree less than or equal te 2. The construction of examples of Calabi-Yau three- 
folds has evoked interest in recent vears. One of the iniportant sources of these 
examples is te take «overs of threefolds of minimal degree. In Section 4. we prove 
some existence theorems for Calabi-Yau «overs of threefolds of miuimal degree. For 
instance, frein these results, whi«h are more general, it follows that a Calabi-Yau 
cover of prime degree greater than 3 induced by a complete linear series canner 
corne frein a group action. 
We will expand on these ideas in two forthçoming articles, [GP3] and [OP5]. 
In the first we study the canoui«al ring of higher-diniensional varieties of general 
type whose canoni«al morphisln maps ente a va.riety of minilnal degree. One of 
the results in [GP3] shows that the converse of the theoreln of Ciliberto and Green 
for surfaces proved in this article is false for higher-dilnensional varieties of general 
type. In the second we «arry out a detailed study of homogeneous rings associated 
te line bundles on trigonal «urves. 

1. PRELIMINARIES 

Convention. Tbroughout tbis article we will work over an algebraically closed field 
of characteristic O. 

[11 this section we will recall seine known fa«ts about the push-forward of the 
structure sheaf of a variety by a fiat, finite morphisni. We summarize these fa«ts 
below, and refer for the proof te [HM], Section 2. 
Let X and 1" be algebraic varieties. Let 7r : X  Y be a finite, fiat morphism 
of degree n. \Ve have the following facts: 
1.1. The sheaf Tf.ex is a rank n, locally flee sheaf ou 1- of algebras over Or. 
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1.2. There exists a nlap 
1 
-tr" 7r.Ox  Oy 
of sheaves of Oy-modules defined locally as follows: Given a E rc.Ox, we consider 
the honolnorphisn of Oy-lnodules 
.Ox  .Ox 
indu('ed by umltiplication bv . Then we define tr(a) as the trace of such a 
homomorphisln divided by n. 
1.3. £tr is surjective; in fact, thc lnap Oy  u.Ox induced bv u is a section of 
2tr. Therefore the sequence 
t 
0E.Ox O 0 
split.s. E is the kernel of %tr and locallv consists of the trace 0 elements of 
 will call E the trace-zero module of u. 
1.4. u.(Ox) is a sheaf of Oy-algebras; therefore, it h a multiplicative structure. 
It.s multiplication nmp is an O -linear map 
[O-eE][O-e] O,-eE 
ruade of four components. The first component 
is given bv the multiplication in Oy, and therefore goes to Oy. The components 
OrNE O)-oE, 
E NO)- 
are given by the le[t and right module structure of E over Oy. and therefore go to 
£. Finally, there is a fourth component 
which factors through 
S2 E  0-  E , 
for nmltiplication in .Oy is conmmtative. 

2. COVERS OF SURFACES OF MINIMAL DEGREE 
Out purpose in this section is to study the generators of the canolfical ring of 
certain surfaces of general type. Specifically, we are interested in studying those 
regular surfaces of general t.vpe whose canouical divisor is base-point-free and such 
that the ilnage of the canonical morphism is a variety of minimal degree. We obtain 
the following result. 
Theorem 2.1. Let S be a regular surface of general type witl af worst caonical 
singularities and uch tlat its canonical bundle Ks is base-point-free. Let ç be the 
canonical morphism of S. Let n be the degree of ç and assume that tle image of  
is a surface of minimal degree r. Then: 
1) if n = 2 and r = 1 (i.e., if  is generically 2 : 1 onto p2), then the canonical 
ring of S is generated by its part of degree 1 and one generator in degree 4: 
2) if n  2 or r  1, then the canonical ring of S is generated by its part of 
degree 1. r(n - 2) generators in degree 2 and r - 1 generators m degree 3. 
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The knowledge of how manv linearly independent generators are needed in each 
degree is obtained from the knowledge of the image of the multiplication maps of 
global sections of powers of the canolfical bundle. We study those multiplication 
lnaps by studying similar lnaps of a curve C in IKs]. Thus we will first prove the 
following proposition. 

Proposition 2.2. Let C be a smooth, irreducible curve. Let 0 be a base-poi77t-free 
line buTTdle o C such that 0 °2 = Kv. Let re be tbe rnorpbism induced bg I01, let 
 be the degree of rr. and assume that re(C) is a ratioal normal curve of degree r. 
Let ff(s, t) be the multiplication map 
H°(O ®) ® H°(O 'st)  H°(O'r-+t), for all s, t > 0. 
The codimension of the image of /3(s, t) i H°(O ®s+t) is as follows: 
a) If r = 1. the codimension is: 
a.1) n- 2. for s= t = l, 
a.2) 0, for s = 2, t = 1. i.e., 13(2, 1) surjects, 
a.3) 1, fors=3, t= 1, 
a.4) 1, fors=t=2, n = 2 and O if n >2. 
a.5) 0, for s >_ 4, t = 1, i.e., p(s, 1) surjects for all s >_ 4. 
b) If r > 1. the codirnensio is: 
b.1) r(n - 2). for s = t = 1, 
b.2) r-l. fors=2, t=l, 
b.3) 0. for s > 3, t = 1. i.e., ff(s, 1) surj«cts for all s > 3. 
Moreover, if r= 1 and n = 2. then the image of/3(2, 2) and the image of/3(3, 1) 
are equal. 

In order to prove Proposition 2.2, we will use the following. 
Lemnla 2.3. Let C. 0 a'nd re be as in the staterneTt of Propositio 2.2. Then 
rc.Oc = Op, ( (n -- 2)O1-, (-r -- 1) ( Op, (-2r -- 2) . 
Pro@ Since the ilnage of rr is Slnooth and of dilnension 1, rr is fiat. Then îr.Oc = 
Op @ E as Op,-lnodules, with E a vector bundle over P of tank n - 1. We noxv 
show that 
e = (n- 2)Ol:,,(-r- )¢, Ol:,,(-2r- 2). 
We have rr.0 = rr.Oc x Op,(r) and rr.tçc = rc.Oc ® Op,(2r), bv the projection 
formula. Any vector buudle over P splits: hence 
g.O C = Op (D E = Op (90p (al) -'- @ (._9pa (an-l)  
for solne negative integers al,..., an-1 (C is connected). Then hi(Içc) = 1 ilnplies 
that exactly one of the ai's, let us say an-i, satisfies an-1 ÷ 2r = -2. On the 
other hand, since re is induced by the complete linear series ]0[, h°(O) = r + 1 = 
h°(Op (r)); so ai + r _< -1 for ail 1 < i < n - 2. Filmlly, since the degree of 0 is 
g(C)-l, we have bi(0) = h°(O) = r+l. Sillce hl((,Qp (-r-2)) --- r+l, ai+r >_ -1 
for ail 1 < i < n - 2, and so ai ÷ r = --1 for ail 1 < i < n - 2. [] 

(2.4) Proof of Proposition 2.2. In Lellmm 2.3, we have COlnpletely deternfined 
the structure of r.Oc as an Opt-lnodule. Now we look at the structure of rr, Oc 
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as an Op,-algebra. If ,t = 2. it is completely detelmined by the t)ranch diviser of 
7r on P. since in this case rr is cyclic. If 7t > 2, we observe the following: 
For seine 1 < i.j < 7 - 2. the projection of the map 
(2.4.1) Op1 (ai) @ Op, ((tj)  7r.O C te Op, (-2r - 2) 
is surjective: in fact. it is an isolnorphism. 
This is se because otherwise Op1 .) Op1 ((/1) t "'" Ç' Op1 (Ort--2) would be an 
integra.1 subalgebra of rc.Oc, free over Op of rank , - 1. Then n - 1 should divide 
7. which is net possible if 7 > 2. 
New we will use ont knowledge of Tf.ex- te stndv the lnaps/3(s, r) which appear 
in the statelnent of the proposition. We will write /4 iii place of /3(s. 1). Let 
R = H(O). Then, since 0 = Tf*Op, (r). bv the projection formula. 
RI = H0(OpI (r)). 
nl ---- H°(Op, (lr))  (, - 2)H°(Op, ((1 - l)r - 1)) ( H°(Op, ((l - 2)r - 2)), 
/1+1 = H°(OP l (Il n u 1)r)) @ (,, - 2)H°(Op, (lr - I)) @ H°(Opl ((l -- 1)r - 2)). 
Therefore an element of Rt. i.e., a global section of H°(O"l), is a sure of n com- 
I)onents, one in each piece of the al)ove decoml»osition of Rt. On the other hand, 
the product of an elelnent of Rt belongilg te one of the blocks with an elelnent of 
R is determined bv the ring structure of (gp and bv the module structure of 
More precisely, the lestriction of/3t te H°(Op (If)) ® H°(Op (r)) lnaps, in fact 
isolnorphically, ente tt°(Op ((! + 1)r)). The restriction of t3t te each of the blocks 
H°(Op ((l - 1)r - 1)) ® H°(Op (r)) naps te the corresponding H°(Op, (/r - 1)). 
This restriction is 0 if (l- 1)r- 1 is negative and an isomorphism otherwise. 
Likewise, the restrict.ion of pt te H°(Op((1 - 2)7" - 2)) ® H0(Op1 (r)) goes te 
H°(Op((! - 1)r - 2)), being 0 if (1 - 2)r- 2 is negative and an isomorphism 
otherwise. Theretbre it is crucial te tell which blocks of a given R are 0. \Ve have 
I = H°(OP '(r)), 
R2 = H°(Opl(2r)) ( (,, - 2)H°(Op1 (7"- l)). 
and if 1 > 3. 
nl = H°(Op,(lr))  (Tt - 2)H°(Op, ((l - 1)r - l))  H°(Op, ((I - 2)r - 2)) . 
Ail the direct smmnands appearing in the above formulae are nonzero, except 
H°(C)p,((1- 2)r- 2)) when l = 3 and r = 1 and (n- 2)H°(Op,((l- l)r- 1)) 
for all ! and all r when 7 = 2. We new deterlnine the image of fit- If 1 = 1. 
the image of/31 is H°((._gp (2r)), which has codimension (7 - 2)r in R.2. If 1 = 2. 
the image of/32 is H°(Op, (3,')) @ (7 - 2)H°(Op, (2r - 1)), which bas codimension 
r - 1 in Ha. If 1 = 3 and r _> 2 or if 1 _> 4. the image of 
surjects. All this proves a.1). a.2), a.5) and b). If r = 1. the image of/3(3.1) 
is H°(Op,(4r))  (n - 2)H°(Op(3r - 1)). which has codimension 1 in /74. This 
proves a.3). If r = 1 and n = 2. the image of/3(2.2) is H°((._gpl (4r)). which has 
codimension 1 irt /4. This proves the first claire iii a.4) alld the last sentence of 
Proposition 2.2. Finally, if n > 2. recall (sec 2.4.1) that for seine 1 _< i.j <_ 7 - 2. 
the projection of the lnap 

(._p1 (I/i) ("., (._p1 (aj)  Tf.(._ C 
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to Op1 (--4) is sm:jective; in fact, it is an isolnorphism. Then. if 7 > 2, the image 
of ff(2, 2) is all R4. This proves the second part of a.4). 
Remark 2.5. Note that 0  = Içc- Then a proof of a.4), alternate to the one given 
above, can be obtained ffoto Noether's theorem and from the base-point-free pencil 
trick. The way in which Noether's theorem is related to the algebra structure of 
w.Oc is shown in [GP4], where we will give a diflrent., simple proof of this classical 
result for a general curve in M. 
From Proposition 2.2 we obtain the following. 
Corollary 2.6. Let C be a smooth curve. Let 0 be a base-point-free line bundle 
o C such that 0" ' = Kc. Let  be the mohi,sm iduced bg 0], let  be the 
degree of  and assume that n(C) is a ratioml normal curve of degree r. Let R be 
@o H°(o';)" Ten: 
1) if r = 1 ad  = 2. lhc rig R is g«,cratcd byit.s part qf degree 1 and one 
generator in degree 4: 
2) if r = 1 atd o > 2. te 'itg R is g«n«rated bg ifs part of degree 1 
gcterators in d«g've 2: 
3) if r > 1. the ring R is gcerated by its part of degree 1. r( - 2) generators 
i?t degree 2 and r- 1 gcn«rators it degree 3. 
Pro@ To know in what dcgrees we need generators, we look at the maI)s ff(s. t) 
of multiplication of sections. Preciselv the mmfl)cr of generators needcd in degrce 
1 + 1 is th(' codimensiou in R+ of the smn of th(' images of /3(1,1). 
- ., , [W]). In particular.  is gcnerated in degrce less than 
or equal to 1 if  snrjects for all k 2 l. Thus 1) folloxvs from part a) of Proposition 
2.2 and ff'oto the fact that the images of (3.1) and (2, 2) are equal. 2) follows 
likewise ff'oto part a) of Proposition 2.2 (note that in this case ff(2.2) surjects). 
Finally, 3) follows ff'oto part b) of Proposition. 
(2.ï) Proof of Theorem 2.1. The proof rests on Proposition 2.2. The idea is "to 
lift" the generators of R to the canonical ring of S. Let us define 
0 -Ns 0 
(i(s)  (ç2)  u((s.-,+,, - 
and let us denote a(s. 1) as a. As in the case of R, the images of a(s, t) will tell 
us the gencrators of each graded picce of the canonical ring of S. In fact, it will 
suce to prove the following: 
(a) If r = 1 and  = 2. a surjects for all 1  1, exccpt if 1 = 3. The images of 
aa = a(3, 1) and a(2.2) are equal and have codimension 1 in H°(K4). 
(b) If r = 1 and n > 2, a surjects for all 1  1, except if 1 = 1,3. The image of 
al bas codimension n - 2 in H°(K). The map a(2.2) is sm'jective. 
(c) If r  2, a is surjective if 1 OE 3. The image of a has codimension r( - 2) 
in H°(Bç). The image of aa has codimension r 1 in H°( 
- 
Thus we proceed to prove (a), (b), (c). Recall that I = ç(S) is an irreducible 
variety of lnilfimal degree and, in particular, normal. On the other hand, the locus 
of the points of I" with non-finite fibers bas codimension 2. Thus, using Bertini's 
theorem, we can choose a smooth curve C of IKs[ such that the restriction of the 
canonical morphism of S to C is finite (and fiat) onto a smooth rational normal 
curve of degree r. Let us denote by 0 the restriction of I@ to C Bv adjunction, 
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Kc' = 0 @z. Since fçs is base-point-ffee, so is 0. Finally, since HI(O\ 
is induced by the complete linear series [0 I, and therefore C, 0 and  satisfy the 
hypothesis of Proposition 2.2. 
ë prove first thc statements in (a), (b) and (c) regarding the maps t- Consider 
the following conmmtative diagram: 
o( -t HO(h-s)  Ho(K)  Ho(o) 
H°(K t)  H°(Os)  .... s   
o -e ,,o ( ,-et+ x  H o (Oat+x) 
H (K s ) ç >  's  
The rightmost horizontal arrows are smective because Ht(Os) = 0. bv Serre 
duality and by Kawamata-Viehweg vanishing. The left vertical arrow triviaIIy sur- 
jects. The right vertical arrow is the composition of the map H°(A 
H°(O t)  H°(O), which is surjective for all l  I again because H(Os) = O. by 
Serre duality and by Kawamata-Viehweg vanishing, and the nmp t of multiplica- 
tion of global sections on C, studied in Proposition 2.2. Then if foIlows from ching 
°t, t+  H°(O @+) nmps the image of t onto 
the diagram tlmt the mal) ,, "s 
the image of t, and that the codimension of the image of t in H°(O t+) is equal 
to the codimension of the image of t in H 
"s » This, together with Prop 
sition 2.2. a.I.a.2, a.3, a.5 and b. proves the claires in (a), (b) and (c) concerning 
the codimensions of the images of the maps t. 
Thus the only things left fo prove are the claires al)out (2.2) when 
consider the commuttive diagram 
o - Ho(z)  HO(O.») 
H°(Iç') H°(Ks)  H°(Iç2) H (h s )  
H°(Iç 3) ç > H°(Iç 4) » H°(O ) 
The rightmost horizontal arrows are surjective because H  (Os) = 0 and bv Serre 
duality, and by Içawamata-Viehweg vanishing. The left vertical arrow surjects, 
we bave ah'eady proven. The right vertical arrow is the composition of the map 
H 0 -2 0 
(Ks)H (0 )  H°(O)H°(O2), which is surjective because S is regular 
and by Serre duality, and the map (2, 2) of multiplication of global sections on 
C. Then if follows from chasing the diagram that the map 0 
H (I; s )  H°(O ) 
maps the inmge of (2.2) onto the image of ç(2.2), and that the codimension of 
the image of (2.2) in H°(O ) is equal fo the codimension of the image of (2, 2) 
in 0 
H (K s ). On the other hand, we know that the image of (2.2) and the image 
of ç3  ç(3. i) are equal of codimension 1 in H°(Oa). if r = 1 and  = 2. Thus we 
conclude that the inmges of (3, i) and (2, 2) in H°(Iç ) are also equal and of 
codimension i. Finally, if r = 1 and n > 2, ç(2.2) surjects by Proposition 2.2.aA. 
Thus we conclude that if r = 1 and n > 2, then (2, 2) surjects. 
Theorem 2.1 compIements known results on generation of the canonical ring of 
smooth, regular surfaces of general type. Ciliberto and Green (cf. [G], Theorem 
3.9.3, and ICi]) proved that, given a smooth surface of general type with h.(Os) = 
0 and h-s globally generated and ç being the canonical morphism, a sucient 
condition for the canonical ring of S fo be generated in degree less than or equal 
fo 2 is that: 
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(1)  does hot map S generically 2 : 10lltO p2, and 
(2) ç(S) is hot a surface of minimal degree other than linear p2. 
As a corollary of the Ciliberto and Green result and of Theoreln 2.1, we obtain 
the following: 

Corollary 2.8. Let S be a smooth regular surface of general type and such that Ks 
is globally generated. Let ç be the canonical morphism of S. The canonical ring of 
S is 9enerated in degree less than or equal fo 2 if and only if 
(1) ç does hot map 5; 9enerically 2 : 1 onto p2, and 
(2) ç(S) is hot a surface of minimal degree other than linear p'2. 

3. EXAMPLES OF SURFACES OF GENERAL TYPE 

bi this section we construct some lleW exanlples of surfaces of general type that 
satisfy the hypothesis of Theorem 2.1. The easiest way one could think of producing 
examples would be to build suitable cyclic covers of sm'faces of mininlal degree. 
However, as the next proposition shows, onlv low degree cyclic covers can be induced 
bv the camnlical morphisnl of a regular surface. So we have to employ other means 
to construct these exanlples. 

Proposition 3.1. Let X be a surface of general type with af worst canonical singu- 
larities and with base-point-free canonical bundle. Assume that the complete canon- 
ical series of X restricts to a complete linear series on a general member of the 
canonical series (e.g., if X is regular). Let ç : X   be the canonical morphi.«m 
fo a surface of minimal degree. Let n be the degree of ç. Let I, be a smooth open 
set of Y whose complement bas codimension 2 and let L be a line bundle on U. 
Assume that 
(ç.Ox)lu = Ou @ L - @... @ L'='-' 

Then n = 2 or 3. 

Pro@ Let H be a gelleral hyperplane section of }" contained in U and let C be the 
inverse image of H by ç. Then C is a slnooth irreducible member of [tç.\-[ and H 
is a smooth rational normal curve. By assunlption the morphism 91c : C  H is 
induced by the complete linear series of a line bundle 0. Bv adjunction 0 a2 = tçc. 
Thus C, 0 and çlc satisfy the hypothesis of Lelmna 2.3, and 
((,lE).O'-, z Opl e (n -- 2)Opl (--r -- 1 ) @ Opl (--2'" -- 2) . 
On the other hand, (q)lc).Oc is equal to the restriction of q).Ox to H, and hence 
(Ic).OC : Op I L t-1 ( . . . ( L ri-n, 
where U is the restriction of L to H. The onlv way iii which (ç[c).Oc can have 
these two splittings is when n = 2 or 3. [] 

Corollary 3.2. Let X be a regular surface of general type with at worst canonical 
singularities and with base-point-ffee canonical bundle. Let  be the image of X 
by its canonical morphism X  Y. If Y is a surface of minimal degree and c2 is a 
cyclic cover, then the degree of op is 2 or 3. 
The next proposition also rules out manv possible examples of covers of odd 
degree: 
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Proposition 3.3. Let X be a surfl«e of general type with at u,orst «anoni«al sin- 
gdarities whose «anoni«al divisor is base-point-free. Let ç be a morphism induced 
by a su.bseries of ]Kxl. I]' is generically finite onto a smooth s«roll } C Pv. then 
the degree of  is even. In particulor, there are hot generically finite covers of odd 
degree of smooth rational no,wal scrolls induced bg subseries of Içx. 
Proof. Let f bc a fiber of I and lct C be a scctiou of Y. Lct -d = C 2. Since Y is 
a scroll, its hyperplane section is linearly equivalent to C+ tuf, for some integer m. 
Then ç.x- = *(c + tuf). Then dcg  = (*f). (*C) = (*f). (Içx - mç*f) = 
(ç*f)  (Kx- + ç*f), which is an cven number.  
Now we constru«t some examples of regulm" minimal surfaces X whose canonical 
uiorlhism  maps onto a varietv of minimal degree, and also mention known ones 
relevant t« this Imper. 
The cases when  is a generically finite morphisui and bas degree 2 or 3 have been 
completely studicd bv Horikawa and Konno (see [H1], Theorem 1.6, [H2], Theorem 
2.3.I, [H3], Theorem 4.1 and [Kou], Lemma 2.2 and Theorem 2.3: see also Mendes 
Lopcs and Pm'dini, [IP]). As it t«lrns out, there exist generically double covers of 
linear p2, the Vç, ronese surface, smo,th rational normal scrolls S(a. b) with b  4. 
and cones over ratioual normal çurves of dcgree 2, 3 and 4 and generically triple 
covers of pe (in particular, cyclic triple covers of p2 ranfified along a sextic with 
suitable singularities) and of the cones over rational normal curves of degree 2 and 
3. Horikawa (see [H4], Theorem 2.1) also dcscribes all generically finite quadruple 
covers X  ', wherc X is a smooth, nfinimal regular sm'face, ç is the canonical 
morphism of X, and ) is linear p2. 
The examples of Horikawa and Komlo just reviewed are examples of covers of 
degrce lcss than or equal to 3 of surfaces of nfinimal degree and quadruple covers of 
P. XX now construct three new sers of exmnples of regular surfaces of general type 
that are quadruple covers of surfaces of minimal degree under the canonical mor- 
phism. These examples are 4:1 covers of smooth rational normal scrolls isomorphic 
to the Hirzebruch surfaces F0 and F, and of quadric cones in p3. 
Example 3.4. IIe constct finite quadruple covers X  -, where X is a s,nooth 
minimal regular surface of general type,  is the canonical morphism of X. and )" 
is a smooth rational scroll S(m, m). m  1. 
Let f be a fiber of one of the fibrations of P and let ff be a fiber of the other 
fil»ration. Then " is PI x PI, and it is embedded in p-+l by f + mf'[ or by 

bi and b2 satisfv the following: 
and b2 = 1. 
= nt + 1 . 

If I" is embedded by If'+ tuf I, let a, a2, bi and b2 satisfy the following: 
eitherb =l, b2=2. a=m+l anda2=l. 
or b = 2. b2 = 1, ai = 1 and a2 = m + 1 . 
For i = 1, 2, let Di be a smooth divisor linearly equivalent to 2(aif ÷ biff) such 
that D and D2 intersect at D  D2 distinct points. Those divisors exist because 
bv the choices of a. a2, b and b2, both 2(af + blff) and 2(af + b2f') are verv 
ample. Let X    be the double cover of " ranfified along D. Since D is 
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Slllooth, se is _\-r. Let D. I)e the im-erse image in 3. "r of D2 1)v tgl. Since D 2 is 
snlooth and lneets D 1 ai distinct i)oints, D is also snlooth. Let X  X r be thc 
double cover of .\' ralnified along D.. Since X  and D are 1)oth Slnooth, sois .\. 
Let ç = ¢'1 o .92. Now we will show that .\" is a regular surface of genel'al type, 
that Kx = (*O)-(1), and that ç is induced 1).v thc COlnl)lete canonical stries of 
X. First we find out thc structure of y).(,.9.\- as a module over Or. Recall that 
ç2,Ox = (gx, @ l*(gr(-a2f - bf'). Tllen 
@.Ox - I. Ox' t l. (l* OY(--o2f -- b2ff)). 
Since pl.O\', = Oy @ Ot-(-alf - bf'), then by the l)rojection fornmla we have 
ç.Ox = Or ©O-(-alf - blf')©Ov(-o2f -b2f')©O-(-(al +a2)f - (bt +b.2)f') . 
We sec now that X is regular. Rccall that Hl(Ox) = Hl(ç.Ox). Oto" «boite of 
al, 02, bi and b2 ilnplies that alf + blf' and a2.f + b_f' are both very ample; thus, 
by Kodaira vanishing, 
Hl(Oy(-alf - hlf')) : Hl(Oy(-a2f - h2f')) 
= H'(Or(-(a, + a2)f- (b, + b2)f')) = O. 
ThelL since H  (Or) also vanishes, se de H 1 (ç.Ox) al,(1 H  (C9.\-). \\i, nov COml)ute 
Kx. Since ç2 is a double cover ralnifi(d al(mg D., 

For a similar reason, 

I'\- = (2*(Içx' ® ç(O-(a_f + b2f')). 

IÇx, = yg(/Çy ( Oy(olf -t- blf')). 
Then 
Iç\- = ç*(tç-® O}-((o 1 -1- a2)f + (bi + b2)f')). 
Since/£v = Or(-2f - 2f'), it follows again froln the choi«es of al, 02, bi and b2 
that Kx = ç*Or(1). Finally, to sec that ç is indu«ed bv the «olnplete canoni«al 
linear series of X. we compute H°(Içx). We do the COlllI)lltatiOll iii the case 
Or(l) = Or(f + tuf'). The case OF(l) = Or(tuf + f') is analogons. Sin«e 
/\- = * or ( 1 ), 
H°(tçx) = H°(O-(1)) @ H°(O-((1 -al)f + (m- bi)ff)) 
@H°(Or((1 - a2)f + (m - b2)f')) 
Ç-H°(O'((1 - al - o2)f + (m - bi - b2)f'))  
Again, by the «hoi«es of al. a2, bi and b2, the last three direct bllll18 of the above 
expression are 0. Se ç is indeed indu«ed by the complete canonical series of X. 
Example 3.5. H'e construct finite quadruple covers X  Y, u,here X is a smooth 
regular surface of general type with base-point-fcee canonical bundle. ç is the canon- 
ical morphism of X. and l is a smooth rational scroll S(m - 1, m), m >_ 2. 
Let C0 be the lnininml section of F1 and let. f be one of the fibers. Then l" is F1, 
and it. is embedded in p2, by [Ç0 + mfl- Let al, a2, bi and b2 satisfy the following: 
eitheral =l,a2=2, bl=m+l andb2=2, 
or al=2, o2=l,bl =2andb2=m+l . 
For i = 1, 2, let Di |)e a slnooth diviser linearly equivalent te 2(aiCo + bi f), su«h 
that D1 and D_ interse«t at D1-D2 distinct points. The fa«t that such divisors exist 
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follows from our choice of al, o2, b and b2, which irai)lies that of the linear systems 
of D and D2. one is very ample, and the other is base-point-free. Let X'  " 
be the double cover of }" ramified along D. Since D is smooth, sois X'. Let 
D be the inverse image in X' of D2 by ç. Since D2 is smooth and meets D 
transversally, D is also smooth. Let X  X' be the double cover of X' ramified 
alongD. SinceX' andD arebothsmooth, soisX. Let ç=çoç2. Nowwe 
will show that X is a regular surface of general type, that L'x- = ç*Oy(1), and 
that ç is induced bv thc complete canonical series of X. First we find the structure 
of ç.O.x as a module over Or. Recall that ç2.0.x = O.x-, ç*O-(-a2Co - b2f). 
Then 
Ç. OX = I.OX'  çl.(l*Oy(-a2Co - b2f))  
Since ç.Ox, = Oy ç Oy(-aCo - bf), then bv the I)rojection formula we have 
ç.Ox = O- @ Or(-aCo - bf)  Oy(-aCo - b2f) 
 Or(-(a + a)Co - (b + b2)f)  
Xê sec now that X is regular. Recall that H(Ox) = Hi(ç.Ox). Our choices of 
al, a2, bi and b2 inll)ly that a.( + bf and a2Ço + b2f are both be-point-free and 
big divisors: thus. by Kawanlata-Viehweg vanishiug, 
H(Oy(-aaCo - bf)) = H(Oy(-a2Co - b2f)) 
= H'(Oy(-(a + a2)Co - (b, + b2)f)) = 0 . 
Then, since H(Oy) also vanishes, so does H(ç.Ox -) and therefore H(Ox-). X 
now COHlpUte KX-. Since ç2 is a double cover ramified along D, 

tçx = 2*(tçx' ® ç(O)-(O2Co + b2f)). 

For a similar reason, 

K\-, = 2(tç ® Or(a,Co + b,f)). 

Thell 

Kx = o*(tQ-¢ Ov((a, + a2)Co + (bi + b2)f)). 
Since tçy = Or(-2Co - 3f), it follows from our choice of a, a2, b and b2 that 
Kx = ç*Oy(1). Finally, t.o see that ç is induced by the complete canonical linear 
series of X, we compute H°(Kx). Since 
H°(tQ) = H°(O(1)) ® H°(Oy((1 - al)Co + (m- bi)f)) 
(SH°(O((1 - a2)Co + (m - b2)f)) 
(SH°(OY((1 - al - a2)Co + (m - b - b2)f)) . 
Again, bv the choices of a, a2, b] and b2, the last three direct sums of the above 
expression are 0. So 

Remark 3.6. With the saine arguments, if one allows certain nfild singularities in 
D1 and D, then one tan construct examples of covers of Fo and F with at worst 
canonical singularities. 

Finally, we construct an exmnple of a quadruple cover of a siugular surface of 
minimal degree. 
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Example 3.7. 14"e construct an ca'ample of a mnooth, generically finite, quadruple 
cover X  Z of the quadric cone Z in p3, where X is a regular surface of general 
type whose catonical divisor is base-poit-free, ad y) is its canonical rnorphism. 

Let 1- = F2. Let C0 be the lninilnal se«tion of ) and let f 1)e a fiber of Y. 
Let D1 be a Slnooth divisor on 1, linearly equiva.lent to 2C0 + 6f and lneeting C 
transversally. Let D2 be a Slnooth divisor on 1", lineaïly equivalent to 3C0 + 6f and 
meeting D1 transversally. Su«h divisors D1 an(i D2 exist, be«ause 2C0 + 6f is very 
alnple and 3C0 + 6f is base-point-free. Note also that, silwe (3('o + 6f)  C0 = 0. 
C0 and D2 do hot meet. Let X'   be the double «over of Y along D1. Sin«e 
D is Slnooth, so is X'. Sin«e D1 lueets (7'o at two distiu«t points, the pullba«k C 
of C0 by 991 is  Slnooth line with self-interse«tiol -4. Let D bc the pullba«k of 
D2 by 91. Since D1 and D2 meet tl'ansversally, D. is smooth, and sin«e D2 and 
C0 do hot meet, neither do D and C D. Let L be the pullba«k of 2C0 + 3f by 9. 
Let X  X' be thc double «over of X' ahmg D U C. Sin«e D U C is Sllooth, 
so is X. Let 9 = 991 o 992. Thon 

(3.7.1) 

ç.O\- = ç.ç2.0x = ç.(O\-, @ L,*) 
= CO- e, (.gv(-Co - 3f) @ Ov(-2Co - 3f) @ Ov(-3Co - 6f) . 

Sin«e Co + 3f and 3Co + 6f are big and base-point-flee, bv Kawanmta-Viehweg 
valfishing and Serre duality, Hl(Ov(-Co -3f)) = H(Cgv(-3Co -6f)) = 0. Bv 
Serre duality, H  (C9v(-2C¢-3f)) = H 1 (C9v(-f))* = 0. Then. since H 1 (C9v) = 0. 
X is regular. Arguing as in Exalnples 3.4 and 3.5. we sec that 

(3.7.2) 

tçx = ç*(tç ® Ov(3Co + 6f)) = ç*Ov(Co + 2f) . 

Now we compute H°(Kx). Using the proje«tion fbrmula and (3.7.1) and (3.7.2), 
we obtain that 
H°(Içx) = H°(Ov(Co + 2/)) @ H°(Ov(-f)) @ H°(Ov(-Co - f)) 
(H°(Oy(-2Co- 4/)) = H°(Oy(Co + 2/)) . 

Thus the «anonical morphism of X is the composition of ç and the lnorphisln 
Y ¢- Z C pa, induced by the «omplete linear selies of C0 + 2f. Sin«e ¢ contra«ts 
C'o, the canonical norphism of X is hOt finite, but it is generically finite of degree 
4 onto Z, which is a surface of nlinilnal degree, as we wanted. 
On the other hand, if C' is the pullback of C0 by ç, then C' is a smooth line 
with self-interse«tion -2. Thus the lnorphisln ¢ o ç also fa«tors as ç' o ¢, where 

x LX 

is the lnorphisn from X to its canolfical lnodel X and 

is the canolfical morphisln of X. Thus ç' is an exalnple of a filfite, 4  1 canonical 
morphisln ff'oto a regular surface of general type with canonical singularities onto 
a singular surface of lnininml degree. 
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4. APPLICATIONS TO CALABI-YAU THREEFOLDS 
The results proved in Sectons 9 and 3 bave ramifications for Calabi-Yau three- 
fi»lds. F/ecall that if X is a Calabi-Yau threefold and B is a big and base-point-free 
divisor, a general member of ]BI is a surface of general type. Then the geometry 
and properties of surfaces of general type m'e directly related to those of Calabi-Yau 
threefolds. Concretely, the results we have obtained in Section 2 on the canonical 
ring of surfaces of general t,vpe can be 'lifted'" to achieve analogous results for 
Calabi-Yau threefohls in a way similar to the way in which our study of rings 
of curves Mlowed us to obtain results for surfaces of general type. On the other 
hand. constructing examples of ('alabi-Yau threefolds has attracted the attention 
of gcometers in recent vears. One of the important sources for these exmnples is 
1)recisely to take covers of varieties of minimal degree. Proposition 3.1. Corollary 
3.2 and Proposition 3.3 tell us features of generically finite covers of surfaces of 
minimal degree iuduced by the canonical morphism. We will see how these features 
pass on to generically finite morphisms ff'oto Calabi-Yau threefolds to threefolds of 
minimal degree, and, as a consequence, we will obtain, among other things, that 
many a priori possible examples of Calabi-Yau threefolds cmmot exist. 
We start with thc Calabi-Yau threefold aualog of Theorem 2.1: 
Theorell 4.1. Let X be a Calabi-I'ut ttreefold with al worst canonical sitt9ulari- 
ries. and let B be a big atd base-po#t-free lie buttdle on X. Let ç be the morphism 
ind'uced by IBI. Let  be the de9ree of ç. attd assnne tha.t the image of ç is a variet9 
of mitimal de9ree r. Then " 
1) If  = 2 atd r = 1 (i.e., if ç is 9enericallg 2 1 oto p3), the caonical rit 9 
of X is 9enerated b9 its part of de9ree 1 and on, e 9eerator i de9ree 4. 
2) If  7  2 or r 7  1. the cartoical rit9 of X is 9eerated bg ifs part of de9ree 1. 
r(n - 2) 9erterators in de9ree 2 attd r - 1 9enerators i, de9ree 3. 
Sketch of pro@ The proof follows the saine lines as the proof of Theorem 2.1. Let 
us define 
H°(13 ?) ® H¢'(B '-'-«) "(.____) Ho(B..:+t) . 
and denote 7(s, 1) as "fs- The images of "r(s, t) will tell us the generators of each 
graded piece of the ring tï__ 0H°(Bç»). In fact. it would suffice to prove the 
following: 
(a) If r = 1 aud , = 2, ",/ surjects for all l >_ 1, except if ! = 3. The images of 
"fa = 7(3, 1) and 7(2, 2) are equal and have codimeusiou 1 in H°(B-4). 
(b) If r = 1 and n > 2.7 surjects for all / >_ 1, except if / = 1.3. The image of 
? has codimension  - 2 iu H°(B»2). The map 7(2, 2) is surjective. 
(c) If r _> 2, 7t is surjective if 1 _> 3. The image of 'i bas codimension r(t - 2) 
iu H°(B®). The image of',/ has codimension r - 1 in H°(Ba). 
Now a suit.able h.vperplane sectiou of the image of ç is an irreducible surface t" of 
lninimal degree. It.s pullback by ç to X is a surface S of general type with at worst 
canonical singularities. Moreover, bv adjunctiou B 0 Os = Ks, and the complete 
linear series of B rest.ricts to the complete canonical linear series of S: so ols is the 
canonical morphism of S. Therefore S is uuder the hypothesis of Theorem 2.1. and 
the proof follows verbatim the steps given in 2.7, the role of S there being played 
by X here, the role of C there being played bv S here aud the role of Proposition 
2.2 there being played by Theorem 2.1 here. 
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To how wha,t we mean, we outline how to find the ilnages of the maps t. 
consider the following commutative diagrmn: 
H°(B '») @ H°(Ox)  H°(B 5 ') @ H°(B)  H°(B :: .)  H°(Içs) 
H°(B TM) ç  H°(B ''+1)  H 
The rightmost horizontal arrows are smjective because H  (Ox) = 0, by Serre 
duality and by Kawmnata-Viehweg vanishing. The left vertical arrow triviallv sur- 
jects. The right vertical m'row is the composition of the map H°(B 
0 
H (Iç) @ H°(Ns), which is smjective for all I  1 again because H(Ox) = 
by Serre dualitv and bv Kawamata-Viellweg vanishing, and the map at of multi- 
plication of global sections on S, stndied in (2.7). Then it follows from chasing the 
diagram that the map H°(B c+l) 
 n çl S ) lnaps the image of ?l onto the 
image of al and that thc codimcnsion of the image of or in H°(N/+1) (which was 
equal to the codimcnsion of the image of t in H°(Ol+l)) is cqual to the codi- 
mension of 7 in H°(B:»l+l). This, together with the claires in (a), (b) and (c) 
coneerning the codilnensions of thc ilnages of the maps at proved in (2.7), givcs 
the codilnension of the ilnages of the lnaps 7l in H°(B l+). The clailns rcgarding 
3(2.2) are proved analogously. 
As we did in the case of the canoniçal ring of regular surfaces of gencral type, 
we can characterize when @0 H°(B') is gcnerated in degrce less than or equal 
to 2 using Theorcm 4.1 and results fronl [GP-]" 
Corollary 4.2. Let X be a Calabi- l hu thre@ld wilh at worst canouical singulari- 
ries. Let B be a bi9 a,d base-point-free line budle on X atd let ç be the 
induced by IBI. Thc @o H°(Bl) is generated in degree less than or equal to 2 
if and only if 
(1) ç does hot map X 9eterically 2  1 onto P:" and 
(2) ç(X) is hot a thre@ld of minimal degrce other than p3. 
Pro@ Theorem 4.1 tells us that if ç(X) is a variety of nfinimal degree, then the 
ring @lO H°(Bet) is generated in degree less than or equal to 2 if and only if 
(X) = p3 and the degree of ç is greater than 2. If ç(X) is hot a varietv of 
minimal degree, then in the prooN of [GP_], Theorems 1.4 and 1.7. it is shown that 
HÙ(B .:.' ) @ H°(B 1)  H°(B l+l) 
surjects if/ 2. 
The study of the generators of the ring @lO H°(B-l) is closely related with 
the question of when B3 is normallv generated when B is mnple. ecall that a 
line bundle L is said to be normally generated, or to satisfy property ç. if it is 
very mnple and the image of the morphism induced by LI is a projectively normal 
variety. This is equivalent to the ring @0 H°(Le) being generated in degree 
1. In the present context (X a Calabi-.u threefold and B an ample and base- 
point-free line bundle on X), the answer to the question of when B m is normallv 
generated is partially known. If the image of X by the morphism ç induced bv 
IBI is hot a variety of minimal degree, the authors gave a complete answer to this 
question for m 2 2: in this case, in [GP2], Corollm'v 1.1, Theorem 1.4 and Theorcln 
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1.7, they proved that B ®mis normally generated if rn _> 2. The way of proving 
those results serves to illustrate the relation between the study of the generators of 
=o H°(B®) and the nornlal generation of powers of B. For instance, in order 
to prove the llOrlnal generation of B 'v2, the first step is to show that the map 
H°(B e) ® H°(B :')  H°(B 4) 
surjects. This was pr,,ve(l in [GP2] by showing that @ï--0 H°(B»t) is generated in 
degree less thml or equal o '2 and that the mai» 
H°(/3 ®3) ® H°(/3) -- H°(B 4) 
surjects. If (X) is a vm'ietv of mininlal degree, the situation is more conlplex. If 
m > 3. the authors also gave in [GP2], Corollary 1.1 and Theorem 1.4. a complete 
answer: B' is normally generated if and onlv if m > 4 or if m >_ 3 and ç does 
not map X 2  1 onto linear P:t. If m : 2, mlsv«ering the question will settle the 
following 
' 
Conjecture 4.3 (cf. [(.P-I, Conjecture 1.9). Let X be a Calabi-lau threefold and 
let B bc an ample and base-poiat-free line bundle. Then B  satisfles propert!/ No 
f and onl!/ if there is a smooth non-h!/perelliptic curve Cin lB ® Osl, for some 
s  I1. 
This çonjecture would also give a characterization of when B ®2 is very ample. 
This question of v«hen B 2 is very ample is also open. 
One might ask what light the results proved in this section shed on the conjecture. 
Theorem 4.1 says that if (X) is Pa. then the ring @ï--0 H°(B®) is generated in 
degree less than or equal to 2 if and Olfly if the degree of  is greater than 2. hl 
this case, however, it was seen in the proof of Theorenl 4.1 that ,a did hot surject. 
On the other hand. if ç(X) is a variety of minimal degree different from Pa, ,a 
does surject but t0 H°(B®t) is hot generated in degree less than or equal to 
2. Therefore the strategy outlined belote to studv the normal generation of B ®2, 
which xvorked when (X) was hot of nlilfimal degree, does hot work if c2(X) is of 
minimal degree. We point out that the conjecture is nevertheless truc if (X) is 
linear Pa (sec [GP2], Corollary 1.8). This also follows from the methods of this 
article, by studying the map ,(2.2) in the proof of Theorem 4.1. Thus the onlv case 
left in order to settle the conjecture is when (X) is a variety of minimal degree 
different from Pa, which should he addressed using a subtler strategy. 
The results of Section 3 regarding the structure of the canonical morphisms of 
regular surfaces of general type onto surfaces of minimal degree has some interesting 
consequences for Clabi-Yml threefolds. As ve will see, Proposition 3.1. Crollary 
3.2 and Proposition 3.3 prevent many a priori natural examples of Calabi-Yau 
threefolds from existing. This also shows that if there do exist examples of prime 
degree Clabi-Yau covers of threefolds of minimal degree induced by complete liuear 
series, then thev cannot corne from group actions. We smmnarize this in the next 
two corollm'ies: 
Corollary 4.4. Let X be a Calabi-Yau hreefold with ai wors canonical sia9ular- 
ities. Let B be a base-poin-free line bundle. Le X  I" be the morphism induced 
bg the cornplete linear series lB[. Let n be he degree off. Let U be a smooh open 
se of I" whose complernea bas codimension 2. and let L be a liae bundle on U. 
Assue hat 
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If t" is a variet.q of minimal degree, then the degree 7 = 2 or n = 3. bt partcular, 
 if  is a c.,clic cover, the degree of ç is 2 or 3 
Proof. Let t" be a suitable hyperplane section of I', and ,b' the lmllback of t" by ç. 
Then ç[s is the canonical nol-phisln of S and satisfies thc h.vpothcsis of Proposition 
3.1, and the thesis is clear. [] 
Notation 4.5. We will call a lllOli)hislll  sa.tisf.ving (*) iii the statcment of Corollary 
4.4 a quasi-cyclic cover. 
Corollary 4.6. Let X be a Calabi-lau threefold with at worst caTonical sin9tdar- 
ities. If X --, l" is a 9enericallg finite mo7»hism oTto a smooth scroll l" C pN, 
then the degree of ç is even. In particular, there are no generically finite covers of 
odd degree of sznooth rational normal scrolls. 
Pro@ The proof is analogous to that of Corolla.ry 4.4. using now Proposition 3.3. 

hl [GP2] we described what finite lnorphisms frolll ri Calabi-Ym threefold onto 
a variety of minimal degree induced bv complete linear series were possible. After 
Corollary 4.4 and ('orollary 4.6 we can obtain the following shm'per version of the 
result in [GP2] (compare with [GP2], Proposition 1.6): 
Proposition 4.7. Let X be a smooth Calabi-lau threefold, let ç be the morphism 
induced bl the complete linear series of an ample and ba.se-point-free line bundle B 
on X. and let  be the de9ree of ç. If ç(X) is a varict'g of miimal de9ree, then 
one of the .[ollowin 9 occurs: 
1) t'=P3 andn<_24. 
2) t is a smootb q.uadric hypersurface izt p4 and n = 2, 4.6.8, 10.12 or 14. 
3) t" is a sznooth ratioTtal normal scroll of dimension 3 in P5 and  = 2, 4.6, 8, 10 
or 12. 
4) Y is a sm.ootb ratioTtal normal scroll of dimension 3 i7 pN, - _> 6. n = 2. and 
X is flbered over P with a smooth K3 surface as a general fiber. The restriction of 
B to the general flber of X is hgperelliptic, witb sectional genus 2. and its complete 
linear series m.aps the fiber 2  1 onto a general fiber of the scroll. 
5) Y is a smooth ratioTtal normal scroll of dimension 3 in PN N >_ 6. n = 6. 
and X is fibered over p1 with a smootb abelian surface as a general fiber. The 
restriction of B to tbe general fiber of X is a (1.3) polarizatiom and its complete 
linear series maps the fiber 6 " 1 onto a general fiber of the scroll. 
6) Y is a cone over a cozic in pe. 
7) Y is a coTe over a twisted cubic in p3. 
8) Y is a cone over a Iéronese surface. 
In addition, if X bas at worst canonical singularities aTd ç is a quasi-c!Iclic 
cover, then n = 2 or 3. 
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ABSTRACT. A chain order ofa skew field D is a subring // of D so that d E D\R 
implies d -1 ( R. Such a ring R has tank one if J(R), the Jacobson radical of 
Ff, is its only nonzero completely prime ideal. We show that a railk one chain 
order of D is either invariant, in which case R corresponds to a real-valued 
valuation of D, or R is nearly simple, in which case R, J(R) and (0) are the 
only ideals of R, or R is exceptional in which case R coIltains a prime ideal Q 
that is not compleIely prime. \Ve use the group ,¥1(H) of divisorial R-i(leals 
of D with the subgroup 7-tIR) of principal R-ideals to charactcrize t hese cases. 
The exccptional case sul»(livides further inIo infinitely IIlany cases depending 
on the index k of 7-/(R) in J/(R). Using the covering group lE of SL(2, N) and 
Ihe result that the group ring TlE is einbeddahle into a skew tield for T a 
skew field, examples of tank one chain or(lers are const ructed for each possible 
exceptional case. 

INTRODUCTION 

A sul)ring R of a skew fit,ld D is called total if d in D and d llOt iii R ilnl)lies 
that the inverse d - is contained in R. It tbllows that for such rings 5' the lattice of 
right ideals as well as the latticc of left ideals is linearlv ordered bv inclusion: 5" is a 
chain dolnain. Conversely. ai13" chain donlaill  is ()re and is a total subring of its 
skew field of quotients D. The total subrings of fields are exactlv valuation rings. 
corl'eSpollding to valuation fUllctions iiito lilmarly ordered groups. In particular, if 
we take nontrivial subgroups G of the additive group (IR, +, <_) of the reals as value 
groul)s , then we obtain the COlnnmtative vahmtion rings of ralik one. Such a ring 
Cml also be characterized as a niaxinlal subring of a field, or as a valuation ring with 
exactly one llOllZ(ro prilne ideal, hl the iion-commutative case we llillSt distinguish 
between prime ideals and coinpletely prime ideals: An ideal B - 5" of a ring 5" is 
prilne if Ii I.). C_ B ilnt)lies I C B or I _C B for ideals I and I of R. If ab  B 
iniplies a  / or b  / for elements ri, b iii 5". then / is called completely prime. 
A total subring R of a skew field D will be called a chain dolnain of rank Olm if 5" 
has exactlv one llonzero COlnpletely prime ideal. This ideal will thell be J(,R), the 
Jacobson radical of R. 
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We prove iu Theorem 1.9 that a rank one chain domain /? is either invariant, 
i.e., all one-sided ideals are two-sided, or itis nearly silnple in which case H, .l(H), 
and (0) are its only ideals, or /ï' is exceptional in which case /? coutains a prime 
ideal that is not completely prilne. The exceptioual rank one chain domains are 
classified further with the help of the group Ad(H) of divisorial R-ideals and the 
subgroup 1-/(/) of Ad(/?) of principal/?-ideals. The lattice of two-sided/?-ideals is 
then determined by the index k of 1-/(/?) in Ad(/?), and we soEv that/? is exceptional 
of type 
These results are proved in the more general case of coims P in groups G where 
a cone P of G is a subsemigroup of G so that g E G\P iInplies 9 -1 E P. 
That rank one chain doInains are either invariant, nearly siinple or exceptional 
was proved in [4]. Invariant rank one chain orders of D correspond to valuation 
fimctions from D* into (IR, +, <_). Nearl,v simple chain domains were constructed 
in [8], [16], [5] and [3]. The construction of exceptional rank one chain domains, 
however, appeared to be elusive even though Posner in [19] hinted that such riugs 
might exist, and the classification of hypercyclic rings by Osofsky in [18] is complete 
only if su«h rings do not exist. I. N. Herstein had considered the problem and this 
existence problem was also encoulitered in [14]. Ve construct in this paper excep- 
tional rank one chain domains of anv type (Ce): TheoreIn 4.4 and Crollary 4.6. 
We do this by first constructing exceptional cones Pe of type (Ce) in subgroups 
H of the universal covering group G of SL(2,1R), Theorem 3.8, and then apply 
Dubrovin's result in [11], where he constructs ail exceptional rank oIm chain ring 
of type (C1) associated with a cone 

1. CHAIN DOMAINS AND CONES 

1.1. Basic properties. A ring R is a right chain ring, if the set of all right ideals 
of R is linearly ordcred with respect to inclusion. Left chain riugs and chain rings 
are defined silnilarly. A chaiu dolnain R has a classical skew field of quotieuts D 
aud can therefore be considered as a total subring of D ([7]). 
A subsemigroup P of a group G is called a cone of G if G = P U p-1 and P 
is a pure cone if in addition P f3 p-1 = {e}. There is a close commction between 
cones Pinagroup G and right orleft orders: if Pis acone of G and a,b  
then _ < defined by a _ < b if and only if a-lb  P defines a left preorder, and a<_b 
if and only if ba -  P defines a right preorder on G. The relations "<_" and 
 ê 
are right orders and left orders on C, respectively if and only if the cone P is pure. 
Fiually, if P is pure, then the right order defined by P agrees with the left order 
defined by P if and ouly if aP = Pa for all a in C,, i.e., P is invariant under all 
imper automorphislns of C,. The group C, is then linearly ordered. 
Let P be a coue of a group G. A nolmmpty subset I of G is called a left P-ideM if 
PI c_ I and I c_ Pe for a suitable elemeut a in G. The second condition is satisfied 
for anv I ¢- C, provided I satisfies the first condition. If in addition I C P, we say 
I is a left ideal. Flight P-ideals, P-ideals and right ideals and ideals are defined 
similarly. An ideal B of P is called a prime ideal if B -¢ P and aPb C B inplies 
a  B or b E B for a, b  P. If ab  B implies a  B or b  B for the ideal B -J= P 
of P, then B is called completely prime. 
We collect elementary properties of a cone P in G. We tan assume that P -J= G. 
Let U(P) = P fil p-l, the subgroup of units of P. 
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a): J(P) = P\U(P) is the nlaxilnal right and the nlaxilnal left ideal of P: it 
is the Jacobson radical of P and it is a colnpletely prime ideal of P. 
b): The set of right (left) P-ideals in G is linearly ordered with respect to 
inclusion. \Ve define I _< 12 if and ouly if I _D I2 for right P-ideals I and 
I» 
To sec this, one considers fil'St principal right P-ideals oP and bP in G. Then 
either a-b  P and bP C aP or b-la  P and oP C bP. If 1.2  Il, then there 
exists a in I\I and Il C oP C I., follows. 
c): There is a one-to-one corresi)ondence between the set of COlleS P' ¢ G in 
(7, that contaill a COllP P and the set of completely pl'illlP ideals B of P. 
Pro@ Let P ç P' C (7 be concs in G. Then j' ¢ J(P') and j'  P implies j,-1 ¢ p, 
a contradiction. Hence, J(P') Ç J(P) and P' = P U (P\J(P'))-. Converselv. if 
B C J(P) is a completely prime ideal in P, then P' -= P U (P\B) - is a cone 7 (7, 
in G. [] 

d): Let I be an ideal in P with I 7 P and Q = Ç]I '' 7  0. Then Q is a 
«olnpletely prime ideal. 

Pro@ If c G P\Q and ca G Q for somc o in P, then there exists n0 with c  
I'°. However, for any n there exist ai, bi G I with ca = a...a,ob.., b,,. Then 
a...a., o =cdforsonledin Pand a=db...b, G I' follows. Hcnce, aG Qand 
Q is a completely prime ideal. [] 

e): A P-ideal I will be right principal and lefl principal if and only if I = 
zP = Pz for some z ¢ G. 

Pro@ Let I = zP = Pz with zl,z  G. Then z = zla, z = bz fol some 
a, b  P. Hen«e, bza = z. Since I is an ideal, there exists b' in P with bz = zb' 
and z = zb'a follows. Therefore, b'a = 1 and a  U(P), and Pz = zP = ZloP = 
z P. [] 

f): Let P be a cone in G. The set "H(P) of all principal P-ideals of G forms 
a group with ideal multiplication as the operation. "H(P) is isomorphic to 
a subgroup of (IR, +, _<) if J(P) is the Olfl.v completely prilne ideal of P. 
Pro@ If I = zP = Pz and I = zP = Pz, then ltI2 = zPz,P = zzP and 
(zp)- - - 
-- ,, P. It follows that 7-/(P) is a group with P as identit.v. To prove the 
second statelnent let P D zP = Pz. Then Ç](zP)  =  since otherwise Ç](zP)  is 
a completely prime ideal - J(P) by d). "H(P) is therefore an ordered Archimedean 
group and the statement follows from H61der's Theorem (see [13]). [] 
g): A right P-ideal I is a principal right P-ideal if and OlflV if IJ(P) ¢- I. 
Pro@ If I = zP, then zP D IJ(P) = zJ(P). Conversely, if I is hot principal as 
a right P-ideal, then for a ¢ I there exists b G I with «P C bP,  = bj  IJ(P), 
j ¢ J(P), and IJ(P) = I. [] 
We single out cones with the property in f)" 
Definition 1.1. A cone P of a group G ha.s tank one if J(P) is the onlv completely 
prime ideal of P. 
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It follows flore the definitions that a subring R of a skew field Dis total if and 
only if the semigroup R* = (R\{0},-) is a cone in the group D*. 
This relationship between a cone in a group and a chain domain is generalized 
in the next definition. 
Definition 1.2. A total subring R in a skew field D is said to be associated with 
a cone P in a group Gif the following conditions hold: 
i): G is a subgroup of D*, the multiplicative group of D. 
ii): Ever.v element d in D* tan be written as d = gal = 292 with g.g2 in 
G and Ul.«2 in U(R) so that PglP = PgaP. 
iii): RC G = P. 
We also say in this case that the cone P is associated with the chain domain R. 
Proposition 1.3. Let the total subrbtg R of the skea, field D be associated with 
the coe P of the groap G. Then: 
i): Io --> loR defitcs an isomorphimn front the lattice of right P-ideals fo the 
lattiçe of nonzero right R-ideals. The im,erse of this rnappmg assigns I C G 
lo the nonzero right R-idcal I. 
ii): The correspmMeucc dcfincd in i) preservcs the properties of being an ideal. 
a completel9 prive ideal, a prime idcal, and a principal right ideal. 

Pro@ i) If I0 is a right P-ideal. then tvo 
form a = 9u, b = g2tt2 for gi  I0 and ui  
and gl = gP. P G P follows. Therefore, a 
that loR is a right R-ideM. since gI C_ P 

nonzero elements a. b in loR have the 
U(/). We can assume that g P C_ g2P, 
-t--b = g2(ptq -t--"a.2)  /oR: this shows 
C_ R for some g  G C D. Further, if 

g  IoRçG for a 
It follows that hg  
Similarly, one can 
(I n G)R = I. 

right P-ideal Io, then 9 = 
¢I0anduCU(R) GG= 
show that I VI G is a right 

hg'u for h  L» g' Ç P and n  U(R). 
U(P); hence, 9  Io and loftG = Io. 
P-ideal if I is a right R-ideal and that 

For ii) we only show that the right P-ideal I0 is a P-ideal if and only if I0R is 
an B-ideal. Let r  R and h  Io, a P-ideal. Then r = plu for p  P,   U() 
and rh : plulh = pL'u2 tbr ulh = L'u2 with u2  LT(B) and h.k  G. Bv ii) of 
Definition 1.2 we have PhP = Pi'P; k  Io %llows and rh  Io, which shows that 
I0B is also a lefl B-module and then an B-ideM. Comrsely. if IoB is an B-ideal 
%r a right P-ideal 10, then I o = IoB  G is a P-ideal. 
Some variations of the results in this section tan be round in [12] and [6]. 
1.2. Divisorial ideals.  consider certain P-ideals for a cone P which will fornl 
a group in case P has tank one. 
Definition 1.4. Let P be a cone in a group G. The divisorial closare Ï of a right 
P-ideal I is the intersection of all principal right P-ideals containing I : 
ï: 
hPDI 
A right P-ideM I is called divisorial if Ï = I. 
If we replace the cone P by a total subring R. we obtain the definition of the 
divisorial closure of a right R-ideal and of a divisorial right R-ideM. In addition. 
we assume that a divisorial right H-ideal is nonzero. 
b collect a list of properties: 
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Let P be a COlle in a group G, I a P-right ideal. Then: 
a): Ï_D I; 
b): I = I: 
c): gI = gÎfor any g in G: 
d): I is non-divisorial if and only if J(P) is hot a principal right ideal and there 
exists ail elelnent z in G with Î= zP and I = z,l(P). If, in addition. I is a P-ideal 
and rank P = 1, then Ï= zP = Pz and I = zJ(P) = J(P)z. 

The properties a, b, and c follow directly froln the definition. To prove d) we will 
write J instead of .l(P) and assume that ï D I aud that z C Ï\I. Then î  zP D I 
and Ï = zP follows; then I = z J, since zjP  I for some j  .I(P) leads to a 
contradiction. This also shows that .1 is hot a principal right ideM. If J is not a 
principal right ideM, then cP  zd implies z-«P  d and z-«P  P, cP  zP 
for c,z  G. This means that Ï = zP for I = zd and hence Ï D I. If zP is a 
P-ideal, then certainly zJ is a P-ideal. Conversely, if zJ is ail ideal, tllen zP is an 
ideal, since otherwise there is an a  P and a j  .l with azj = z, a contradiction. 
Finally, we asslillle that Ï # I and I is a P-ideal alld that P has rank ont'. Tllen 
ï = zP and the left order O((I) = {.q e G J gï ç ï} # G «ontains the cone P as 
well as the cone zPz - both of which are maxinlal. It fifllows that P = zPz -, 
Pz = zP = Ï and .lz = z,l = I.  
b list a property that was proved in the proof of d): 
e): I is a P-ideal if and oldy if I is a P-ideal. 
The next result shows that in the correspondence between right R-ideals and 
right P-ideals, divisorial right ideals correspond to each other if the chain domain 
R is associated with the COlle P. 

Proposition 1.5. Let R be a total subrin9 of the skew field D associated with the 
cone P irt a group G. Then the right P-ideal I is divisorial if and only if the right 

/'-ideal I B is divisorial. 
Pro@ Assume I is divisorial, 
with hP D_ I, and 

i.e., I= Ï= Ç hP. Then I/i'ç h/i'for all h cG 
hPDI 
hR. To show the reverse inclusion, let d  ÇI hB 

hPDI 
for hP D_ I and d = hr h = gin fol g  lâ. m  U(/). HellCe, 9 = hrh m-1 ¢ 
hHG = hP and 9  hP = I. d  IB follows. Now assulne that .4 is a divisorial 
right R-ideM. A =  dR. Any Stleh d = gin fbl'g G G, II G U(). Hen«e, 
dRA 
AaG = (dR)G = (gRG) = gP. which shows that 4aG is divisorial 
and A  G is nonempty, since .4 is nollzero.  

For any subset I of a group (7, we define the following three subsets of G : the 
right order O(I) = {9 e G [ 19 C I}, the left order O«(I) = {9 e G [9I ç I}, and 
the inverse I - = {9  G I IgI ç I}. 
It follows that I - = {9 e G 191 ç O,.(I)} = {9 e G I I9 ç O«(I)}. 
We have the following two properties where P is a cone in the group G : 
f): If I is a right P-ideM. then Or(I) is a COlle of G and O,.(I) is an over cone 
of P. Fnrther, I is a right O(I)-ideal and a left Or(I)-ideal. and i-1 is a 
right Or(I)-ideal and a lefl. O(I)-ideal. 
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For a proof we observe that for any g in G either gI C I and g  0(I) or I C gI 
and g-1  Oe(I). The test of the statements follow inmmdiately. 
g): O,.(J(P)) = O«(.I(P)) = P, and j(p)2 ¢ j(p) implies that J(P) = 
zP = Pz for solne z  P. 
The first statcment follows froln Property c) iii Section 1.1 since Or(.](P)) D 
P implies that j-lj(p) Ç .I(P) for some j  .I(P). Hence, .I(P) Ç jJ(P), a 
contradiction that shows Or (J(P)) = P and similarly O/(.I(P)) = P. 
The second statenmnt follows from Propcrty g) in Section 1.1, its left S,Vlnnmtric 
version, and Property e) iii Section 1.1. [] 
Even though one can consider the groupoid of all divisorial P-ideals for a cone 
P of arbitrary rank (see also [2]), we restrict ourselves to the rank one case: 
Definition 1.6. Let P be a cone of rank one. Then .Ad(P) is the set of all divisorial 

P-ideals togetller with the operation " 
Il * I2 = Il I2 
\Ve have the following rcsult: 

* " defined by: 
foL P-ideals 11, I2. 

Theoreln 1.7. Let P be a cone of rak oe in a group G. Tben: 
 ) ./( P) is a linearl!! odered gvup: 
d) The iv«rse of an elemett I in .Ad(P) is 
) (P) is a subgro«p of.Ad(P). 
Pro@ We show first that the operation defined in Definition 1.6 is associative. 
On the set of ail P-ideals we define a relation IL  I2 if and only if Ï = Ï2; this 
is an equivalence relation. 
We are going to show next that for P-ideals Il. 1"2 the following equivalence holds: 
(+) II2 ,-, 1112. 
If I1 = Iî and I2 = Ï, thell (q-) iS trivially true. If 11  Iî, then Ï = zP D 
zJ(P) = Il and J = ,J(P) is hot right principal. Also Ïl = zP = Pz is a P-ideal 
by Property d). 
The equivalence (+) holds therefore if and only if the following equivalence holds: 
(++) JI2 ' PÏ2 = 
Hence, if JI2 = I2, we are done. Otherwise, JI2 C 12 and 12 = Pd follows for some 
d in C, by the left synlmetric version of Property g) in Section 1.1. Since I2 is an 
ideal, we have dP ç Pd, P ç d-lpd and the equalit.v d-lpd = 19 since P has rank 
one. Then dP = Pd = 12, dJ = Jd and JI 2 = Jd = dJ , dP = I2 which proves 
the equivalence (++) and hence also (+) in this case. 
Finally, we must prove (+) if I1 = Î1 and Ï D 12. Then, as above, ï2 = aP = 
Pa D aJ = Ja = I2 for sortie a in G. The equivalence (+) then holds if and only 
if the equivalence I1J -- Ï1P = 11 holds. Using the right symmetric version of 
arguments used in the proof of (++), one shows that I1J ' I. This proves (+). 
If 11 -" I and 12  I for P-ideals I, I, 12, I, then 1112  Ïl Ï2 = 1112' g'  ILI .' ' 
Hence ./bi(P) is a factor lnonoid of the nlonoid of ail P-ideals, and the operation  
given in the definition for .Ad(P) is associative. 
Next we show that II - = P for I a P-ideM, and I-I = P follows from similar 
arguments. Since I is a P-ideM, 1-1 is a P-ideal. 
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If I1-1 = P, we are done; otherwise I1-1 C_ J(P) = J. If I1-1 C_ Pz C J for 
some z E J, then II-lz -1 C P and I-lz -1  1-1 which ilnplies z 
P, since P has rank one. This is a contradiction since z E ,1, and 11-1 = ,1, ,1 7  Pz 
for all z E P remains as the only l»ossibility to be considered. It then t'ollows ff'oto 
Property g) that ,l 2 = J, J is hot a principal right ideal, and hence I1-1 = ,Ï= P. 
In order to complete the proof of o) and /) we show that I - is a divisorial 
P-ideal for I a P-ideal. If on the contrary, I - = zJ C zP =- 1-1 and J is hot a 
principal right ideal, then zJI C_ P b.v the definition of i-1, and l»y (+) it. follows 
that zJI C_ P = P. Since zJ = zP, we obtain z.l C_ zÎc P, and hence z E I 1 = zJ, 
a contradiction. 
This shows that .Ad(P) is a group and thal /3) holds. For 11 _D I2 in .Ad(P) 
we define I < I. and .Ad(P) then is a linearly ordered group with P as identity. 
Elelnents in ?-t(P) have the form I = zP = Pz for some z in G with zP = 
and (zP) -1 = z-lP = Pz-1; see f) in Section 1.1 and ï/) follows. This proves the 
theorem. [] 
Corollary 1.8. Let P be a cone of ratk one in a group G. Thet ,(P) and (P) 
are A rch ira edean gro ups. 
Pro@ Let B C P be a divisorial idcal. IfB C J(P) = Jor B = J ¢ J2, then 
ÇB n = 0 by Pl'operty d) in Section 1.1. If J = ,I ', we have ,I = P and henee 
Ç] B' = 0 in all cases, and B'+ C B'. Then ,+1 C_ B', since there are no further 
right ideals between B'+1 and /,+1. This illlplies Ç/' = O, and it follov, s that 
.Ad(P) and 7-{(P) are Archimedean: see also Property f) in Section 1.1. [] 
Related results can be round in [12] and [2]. 
1.3. The classification of tank one cones. The groups fld(P) and ?-/(P) will be 
used to classify tank one cones P in groups G based on the lattice of their ideals. 
In the following theorem and proof we will write J instead of J(P). 
Theorem 1.9. Let P be a cone of rank one in a group G. Then exactly one of the 
following possibilities occurs: 
A) - The cone P is Archimedean. i.e., aP = Pa for ail a in P. IVe distinguish 
two possibilities in this case: 
Al): fl//(P) = -(P)  (Z, +, _<). tt,hi«h is exactlg the case when J ¢ J Then 
every P-ideal is a power of J and the cone is called discrete. 
A2): .Ad(P)  (IR, +, _<) and ?-t(P) is a dense subgroup of ,M(P). 
B): The cone P is nearly simple; i.e., J is the only proper ideal in P. In this 
case 34(P) = (P) = {P}. 
C): The cone P is exceptional; i.e., there exists a prime ideal Q in P that is not 
completely prime. Then: 
i): There are no further ideals between J and (2. 
ii): The ideal Q is divisorial and .Ad(P) = gr {Q} is an infinite cyclic group. 
iii):  Q" = O. 
iv): There eists an integer k >_ 0 such that "H(P) = gr{Qk}. The cone P is 
said fo be of type (C) in this case 
If P is of type (C0), then 
 " D (Q)-I D...DQ - DPDJ DQDQ  DQ 3 D-.- 
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is the chain of P-ideals. 
If P is of type (CI), then 
-" D Q-2 = z-2P D z-2J D (2 -1 = z-IP D z-l.] D P D J D zP 
= Q D zJ » z2P = 0 2 D z2J D ... 
is the chain of P-ideals. 
If P is of type (Ck). k > 2. then 
"'" D (Qk+I)-I D z-IPD z-lJ D (Qk-1)-I D"" 
D Q-1 D PD J D Q D Q" 
D ... D D zP D zJ 
= 0 h" D Qk+l D "'" D Q2k-1 D z2P D z'J 
= (-2 2t D Q.t+l D "'" 
is the «hain of all P-ideals. 

Pro@ If J is the olllv proper ideal of P, then P is of type/3. 
Otherwise, let Q = U I be the lllliOll of ideals of P properly contained in J. If 
.1 : = J and J D Q, then P is exceptional: for ideals I D Q and I2 D Q in P we 
ha'ce If . I2 D_ j2 = ,1 D Q and Q is a prime ideal of P. hot completely prime and 
no fmther idem exists between ,I and Q. The divisorial closure Q of Q is ail ideal 
that CallllOt be eqllal to ,1, since J would then be right i)rillcipal. Hen«e, Q = Q 
is the sma.l]est positive element iii the linearly ordered Archimedean group ,M(P), 
and .Ad(P) = gr{Q} is ail infinite cy«li« group. The subgroup (P) has therefore 
the form (P) = gr{Q t} tbr sonle k _> 0: we sav that P is of type (Ck). 
We can now des«ribe the P-ideals in ea«h case (C) if we recall (sec Property 
d) in Section 1.2) that ail ideal I is either divisorial or of the form c,l = Je with 
Ï = oP = Pe. some c ¢ G and J = J-. It will also follow fl-Olll the rest of the proof 
that if P is exceptional, then J = j2 and J D ( = U I. where the ideals I are 
properly COlltailled iii J, the prime ideal that is llOt completely prime. 
In the case (C0) there are no principal ideals ¢ P and the group ,l(P) = gr{Q} 
«ontains all P-ideals ¢ J. In the case ((71) the ideal Q = zP = Pz is principal 
and .Ad(P) = (P). In the case (C), k _> 2. the ideal Q is principal. However. 
Qh itself cmmot be principal, sin«e otherwise Q = zP implies QhJ ¢,Qh; hen«e, 
Q,I ¢ Q and Q is principal (sec Property g) in Section 1.1). Hence Qh = zP = 
Pz D Qk = zJ = Jz for an element z in P. 
It remains to consider the case where either J ¢ J, j2 
or J = and J = Q = UI 
for ideals I properly contained iii J. hl this case we will prove that aP = Pa for all 
a in P. If for some a in P the right ideal aP is hot a left ideal, then an element c 
exists iii P with cap D aP and caj = a follows for ail element j in J. Bv assmnption 
there exists an ideal I C_ J with j ¢ I and ["] I n = : we obtain the contradiction 
a = caj = cnaj   Ç]I n . We have Pa ç aP, P c_ aPa -1 and P -= aPa -1 since P 
is of rank one. Therefore, Pa = aP for all a in P and P is invariant. 
If J =/: J', then J = aP = Pa, fol" some a in P, is the smallest positive element 
in the Ar«himedean group .Ad(P). Hence, ,4(P) = (P) = gr{J} is the group of 
all P-ideals. 
If J = j2 and J = Q. then (P) is isomorphic to a dense subgroup of (IR, +, _<) 
and .Ad(P) is isomorphic to (IR, + _<). [] 
If R is a «hain order of rank one in a skew field D. then R* = R\{0} is a cone 
in the group D*. We say that R has type (A). (Al), (A). (B), (C), or (C) if and 
onlv if the cone R* is of the saine type. 
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The next result follows fl'oln Prol)ositions 1.3 a.nd 1.5 and Theoreln 1.9. 
Corollary 1.10. Let P be a coe asso«iated witb the rank oe cbai domaiu R. 
The'n P and R bave the saine type. 
2. THE UNIVESAL COVEmNG GROUP  OF SL(2,) 
2.1. The group SL(2,). By SL(2,) we denote, as usual, the group of 2 x 2 
lnatrices with l'cal entries and deterlnilmnt equal to 1. Then 
U= = 0 a - 

and 

3 = r(t) = Sill t COS t t  ] 

are two particular subgroups of G. Every C|elm,nt s  SL(2, ]) ca.n be written in a 
Ulfiquc wa" as 
s=r(l)l, for r(t) a with 0t <2 and u . 
To prove this clailn, let {e, e2} be the standard basis of 2, the Euclidean plane, 
and let the elelnents of SL(2, ) be the lepl'eSeltatiols of linear trans%l'lnations of 
2 with respect to the basis {ci, e}. For every nonzero vector a   there exists 
a unique element t e [0, 2) with a/llall = el cos t + e2 siu t: we write arg a = t iii 

this case. 
Let t = arg S(el) for the givell elelnellt 8  SL(2, 
SOlne elelnent v. since r(-t)s(cl) = ael for o > 0. 
representation is tllliqlle, Sillce lJ Ç/ ---- { [}. I -- (  ï ) 
2.2. The group G. We are going to constrllct the 

ll) and r(-t)s = u  IL ! fol" 
Hence, s = r(t)u and this 
 the identity of SL(2,1R). 

Ulfiversal coveling group G 

of the group SL(2,1R) in this section. \Ve do this first fOl" the subgroup  bv 
fixing a sylnbol, say x, and bv rewriting the additive group of the rem munbers 

ill lnultiplicative fOl'lll: 
R = {x t I t  ll}; X t'  a "t= = Xt'+t2: x t' 
Then R is a linearlv ordered group isolnorphic to 
R to g with r(x t) = r(t) is a group epilliorphisni with 

<_ x t2 z;îe t l _ t2. 
+, <_). The mapping r frolll 
the cyclic subgroup gr{x 2r } 

as its kernel: r is a cover of the Lie group g. Next we define the covering group G 
of SL(2, IR) as the set G = {xtu I xt  R,   U}, the Cartesian product R × U, 
together with the following operation: If x t Ul, xt2zt2 are two elelnents in G and 
tu = 2rrk + ç for k  Z and ç  [0,2rv), then ulr(ç)u = r(b), in SL(2.1R) for 
u  U, b  [0, 2rr), and the product in G is defilmd as xtlUl -xt2u = 
The mapping r from above can be extended to a lnapping froln G to SL(2, 
by defilfing 
(") = ,-(t). 
We want to prove that G is a group and that r is an epilnorphism from G onto 
SL(2,IR). 
Lemma 2.1. The mapping r is onto SL(2, IR). and if a  b = c for elemerts o, b. c 
in G, then r(c) = r(a)r(b). 
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Pro@ The element a.t u in G satisfies v-(.rtu) = r(t)u for the arbitrarv element r(t)u 
in SL(2, OE); ç is onto. Ira = zUl, b = z»u in G, t = 2k+ç, k G Z, ç G [0.2) 
and if ur(ç)u = r()u.   [0, 2). ui, u  U, then c = .rt++'u and 
= (0)(2 + ç) = ()(), 
which proves the lclnlna..  

Several special cases of the associative law fol" the operation defined for G are 
proved in the next few steps. We can consider R as well as U as subgroups of G 
ad the equations 
(+) .r t . u = ad'u, .r t-'tu=.r+tu, and .rtu-u'=xtzu ' 
fifl|ow. We conç|ude also that .r t  a = .r t  b imp|ies a = b for elements a. b G G. 

Lemma 2.2. For any element g = xtu  G atd any m  Z the product g..'' is 
cqtml fo 

Pro@ We have 7rm = 2rck + 99 with/," G Z, and  = 0 if m is even, and  = 7r if 
m is odd. In both cases ur(ç) = r(ç)u follows, which proves the statelnent of the 
lemma. [] 

Lemma 2.3. For an.q a, b  G and ang integer m  Z the following equalities hold: 
.m. («. b) = (.<"- ). b = . («m. b). 

Pro@ Because of (+) the first equation follows, and we can assulne that a = tt  U 
and b = ..t ¢ R in the second equation. 
It remains to prove the following equality: 
(«m. ,,).  = . (m. ) 
where t = 2k + ç, k  Z, ç  [0, 2) and ur(ç) = r()u' for   [0.2), t  U. 
Then (.rm.u).t = zm++g'U '.  distinguish three cases in order to compute 
the right-hmd side of the above equation. 
In the first case, m = 2k' is even and the equality follows ilnlnediately. 
In the sec(md case, wm = 2wk' + w for some k' G Z and  < . Then 
tf  .T 2(k+k )++  .2(k+k )tf - 3 "+  .2(k+k')++g.gt  
since ur(g + ) = lll'(g)T() = T(g)7"()lt t : T(g + )U t in SL(2, N); the equation 
is proved in this case. 
hl the final case, m = 2k' +  for U G Z and ç  . The right-hand side of 
the above equation is then equal to 
U "  "2(k+kt+l)+-  W(k+kt+l)--+llt  
which proves the lelnlna. 

Lemn,a 2.4. Letu G U andt  (Trm, Tr(m+l)) for sornem  Z. Then u.x t = x t u 
o.' ç t'  (,.(,+ )). 

Pro@ Let t = 2rrk+ç for k G Z, q G (0,2rr). If rn = 2k is even, then  G 
(0, rr); hence sinç > 0. It follows that for any u = (0b_l)  U; the argulnent 
a cos  
/, of //r(ç)(el) = r())ttt(el) is also il, (0, rr) since b = a.rg [( 0 o, b-l ) ( SiYl(t'9 )] and 
a - sin ç > 0. Hence t' = 2rrk + ç G (Trm, 7r(m + 1)) as stated in the lelnma. 
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If rn = 1 + 2k is odd, then t = 2rk + 
with ur(ç) = r()'u; is also negative with the above argmnent; hence,   (, 2) 
and t' = 2uk + 
Theorem 2.5. a):  is a group; 
b): The mapping r is a homomorphism frvm  onto SL(2, ); 
c): The center of G is the i¢nile cyclic group gcnerated by x . 
Pmof. To show that the operation defined for G is associative we consider three 
elements xt'ui  G, i = 1, 2, 3 with ri   and ui   and thc equation 
By Lemmas 2.2 and 2.3 this equation holds if and onlv if the following equation is 
true: 
(x"+ " *+)" x+" =/'+%" (*+%" *+a) 
for integers k, m and n. 
It follows that it is suffiçient to prove (.) only in the case where t, t2, t3  [0, ). 
For g = x«u ' and g2 = x 'u with t, t'  N, ', "   we apply Lenmm 2.1 and 
obtain 

,-(t)' = (,'(ri).1-T(t2),,2)- T(t3),3 

and 

in SL(2,1R) where the operation is associat.ive, and therefore r(t)u' = r(t')u" follows. 
This implies u' = u" and t - t' = 2rrk for sonle "  Z. It remains to show that 
W apply Lemma 2.4 ad obtai z  «z for z  U,   [0, ); z 
«ta = «t3 for   U, t3 e [0, n); u2x ta = xt3 for u 2  U, t 3  [0.); and 
l-t2+t : .çt2'3ï for uï Z ç and t2,3  [0, N). 
Therefore: 

gl = (XttUl " Xt2t2) " Xt3U3 = '{xtt+t'U'l U2] " Z3U3 

and 
g2 XttUl (3:t2t2 :Et3t3) xtltl 3;t2+t'3 t 
:  . :  23 
t+t2 3- Il I 
= " ' "1 23" 
Hence, t = t + t + 3 and t' = ri + t2,3 and therefore 
t-t'= ' 
t2 + 3 - t2,3 = 2k. 
However, t + 3 and t2,3 both belong to [0, 2) and k = 0 and the associative law 
follows for the operation defined for . 
Since  bas e = x°E, for E = ( ï), as the identity and xtu bas u-x -t as its 
inverse,  is indeed a group; this proves a). 
The statenlent b) was proved in Lemma 2.1. It follows from Lenmla 2.3 that 
gr {x  } is contained in the center Z() of . Conversely, if xtu  Z() for t   
and u  U, then an application of Lemma 2.1 shows that r(t)u is in Z(SL(2,)). 
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Ht'nce r(t)u = +r(0), u = ( ï), and t = 7rk tbr some k e Z follows. Therefore 
a-  gr{x}, which proves c) and the theorem.  
See also [1] for the fact that G is right orderable, but hot locally indicable. 
2.3. The representation of the group G. To each element 9 = xta  G we can 
assign the projection v(9) = v(x t u) =   . The nmpping I  G  Aut (, 5) is 
defined as 1(« ) = v(9 xt) for9  G, x t  R. That 1 is indeed an automorphism 
of (R, ) folh)ws from the next result. 
Lemna 2.6. For g  G let l be dçfined as above. Then: 
b): 1 is th« id«ntity mapping if and only if g is the identity element in . 
): T,« .bim« t(') = {g e   (-') = .,-'} i « o *çx -  V, 
wbich is an Ore group. 
d): 1 is an automorphism of (R, ) for «v«ry g  . 
Proof. To prove a) we comlmte v(ggx t) and ,,(ge,(g2x*)). Let g = x t' ai, g2 = 
a:t2u2 for ai  U. Then gig2x  xtux2a2xt xtuixt2x t u for some  
U, '   with u2,r t = s.t' u'. Furthcr, a.t ua.t2+t' u' = xt'+ï u' for ux t2+t' = xFù 
fl,r   U, ï  . It follows that v(gg.2x ) = a: '+ï and that v(gv(g2xt)) = 
v(xt, u.r2+t') = t,+ï; this proves a). 
To prove b), assmnc g = .t u and l(x t) = x t for all t  . For t = 0 it follows 
 and assume that u  ). Then (x) = x 
that t = O. X consider t =  = (ô  
implies that 
r[( l)(ï 1)()] :r(l): . 
, ). Finallv. for t =  we must have 
Hence. b=0and u=(ô . 
arg  = - 
and a  a-a  1 follows: hence, 9  e, the identity in , and b) follows. 
To prove c) we observe that st(x°) = {x tu, G  ] ç(z0) = wt = } equals 
U. Hence, I(x t) = xtU -*  U. These stabilizers are Ore groups in the sense that 
the group ring TU over a skew field T is an Ore domain. This is true since U is the 
semidirect product of the following two torsion free abelian groups: 
This proves c). 
Finally, we want to prove d). Since l  is an atttomorphism of (R, ), it follows 
from a) that it is enough to show that ç is an automorphism of (R, N) for any 
u  V. Ve show first that z t > z t implies I(z t) > t,Ç( ) which then implies 
that l is one-to-one and order-preserving. Bv Lemlna 2.4 and Theorem 2.5(c) we 
can assume ri, f2  [0. g). It then follows that  - f  (-, ), and in addition 
f -  > 0 if and only if 
( cos t cos t2 
sin(t-¢)= Det,sint sint) >0" 
 hv (t) = (d)  d  [0.) ,d d = g(,('))  [0.). Th, 
sin t, 
Det (u(C°s t, cos t )) > 0, since Det (u) > 0, and, as in the previous argument. 
sin t sin t2 
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de > dl follows. This shows lhat x d2 = l;(a "t2) > ..d = lç(x h) for t2 > ri and 
that IÇ is order-preserving and one-to-one. 
It remains to show that 1 is onto, and by Lemma 2.4 and Theorem 2.5(c) 
it is enough to show that 1" maps the interval [x °,.r '] onto the interval [x °. x']. 
This, however, follows from the fact that l(a "°) = x °, l(z ') = x' and that 1" is 
continuous. [] 

We will prove next a technical result, which will be used several times. 

Lemma 2.7. Let g = ztu  G with t = 7ch + fo and x q  R with t = 7cm +/10 
for k, m  Z and fo, tlo  [0, 7c)..4ssume that (  )  ]I 2 ?.t, ith arg (  ) = tlo. 
Then l(,r h) = x rr(k+m)+t' for t' = arg (r(to)u( ; )). 

Pro@ Bv definition we have that l(a "t' ) = u(9a 31). Further, «{1 = OE.'{L-'I = 
*(k+m)zt°'tza'h° since z  is in the tenter of G by Theorcm 2.5(c). 
By Lemma 2.4 we have ,«xt° = x i  wit h  G U alld ï= arg (ur(tl0)(  )) e [0. g). 
Hence, z'u.r t'° = x'°+tù. ()n the other hand, t' = arg(r(t0)u(;)) = t0 +  
since both t0, ï  [0,). It fillows that 9.r t = a-(k+)+t°+ia and l(3 "ri) = 
.r(k +m) +t' .  

3. EXCEPTIONAL CONES IN THE UNIVERSAL COVERING GROUP 
In this section we construct exceptional cones of type (C) fol" everv k in the 
universal covering group 
We define first two particular elements Wl, tt' in G which will play m important 
role in this construction. The element Wl = ( î) e U C  and r(wt) = a'l tbllows. 
Next we consider the elemcnt ( 
(0, w) and define we   where u = r(-a)( ï) e U; hence, r(we) = ( ï). 
Lemma 3.1. Let b be an elemet in [0, u). The lira ç;»ï(z b) = x °. 
Pro@ We consider the real mmfl»er b.,, with x b = V (xb). Since u,ï = (   ) and 
r(wî) = wî, we can apply Lemma 2.7 and obtain 
bn = arg [( n ) (cosî)] = arg(cosb+2nsinb) 
sin sin b " 
If b = 0, then b = 0 for all n  0 and the result follows. If b  (0. ). then sin b > 0 
and linl (cosb 
 are now read3 to define one of the main objects of this paper: 
e = { e ç [ (.0)  0}. 
The next result shows that this is an exceptional cone of type (C 1) in 
Theorem 3.2. a): ThesetF={ge[ I(x °) Rx°}isaconein u,ithU(F)= 
U. 
b): Any right F-ideal is either a principal right ideal xtF or of the form 
xJ() for some t 
d): The cone P is exceptional of tank one with Q = «P the prie ideal that 
is hot completely prime;  is exceptional qf type (C). 
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Pro@ a) If g and h are eleinents in IP, then th(:r °) = V9(Vh(:r°)) _> ç(x °) OE 0 
bv Lemina 2.6 a) and d), and gk   follows. 
If g is not in P, then lÇ(x °) < x°; hence, x ° < I'- (x °) again by Lemma 2.6. 
and g-   follows and  is a cone of G. It also follows from the above arguments 
that g, g-   implies I(x °) = x ° and g  U. Conversely, U C  and U() = U 
follows. Hen«e, J() = {g e G  I;(x °) > .r°/. 
b) Let I be any right -ideal in G. Then it follows that x  = iiff{ IÇ (x °)  g e I} 
exists since I Cc for soine c  G. çk  will show that Ï = x for the divisorial 
closure Ï of I, see Definition 1.4. By definition we have x  g = xç for ail 
g  I since a N ff; hence x  I. Conversely, if h  G with h = xT  I. then 
7  ç(x°) for ail g in I and 7  a follows; hence Ï = x. It follows that either 
I = z = ïor that Ï= x and I = x%/(P); see Property d) in Section 1.2. 
c) Assulne that zt is a -ideal. For t = m + t0, m  Z and t0  [0,) it 
follows that x° is also a -ideal since x  is central in G. If t0 > 0, it follows 
froin Leinina 3.I that there exists a power wï of w in U with wïxt° D xt°, a 
coixti'adiction that shows that xt = xm. If I = xtJ(P) is a -ideal, then Ï = xt 
is a -ideal by Property d) in Section 1.2, and t = mn by the above arguinent. 
d) We have  D J() D x = Q and Q is hot a coinpletely priine ideal of , 
since z /- x /  Q, but z /  Q. However. Q is a prime ideM, since anv ideals A 
and B of P that contain Q properly, also contain J(); hence. AB  J()J() = 
J() D Q, and it follows that Q is a prime ideal that is hot completely priine. 
There are no firther ideals between J() and Q. and  Q = . It follows that  
is an exceptional cone of type (C1): see Theoreii 1.9.  

We denote by F the subgroup gr{wl,w2} of G generated by u,1 and w2. This 
subgroup is mapped by - onto the subgroup gr { (  î), ( z  ï) } of SL(2, N) generated 
by ( î) asd (z  ï). Since this subgroup of SL(2, N) is free (see [I5], 14.2.I), the 
group F is free of tank 2. 

It follows [roln Lemma 2.7 that 
t' xarg ( -3 ) 
Iïx(x )=x = , since 
and 

,i -1 = ( ï) e u 

t'= arg [(-]z)()] = arg(). 

By a flrther application of Lemma 2.7 we obtain 
I)q(X O) = Vw2 (xarg(  3)) 
with 

_ 2F t" 

t" = arg ['r(wz)(  )] = arg [(91 ï)( - )] = arg(-). 
= x '+g(]), which proves the lemma. 

Hence, I/), (x °) = .r arg ( ) [] 
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In order to construct further cones we consider a, subglotlp H of G that contains 
F and defilm 
PH =Hn?. 
It follows imInediately that PH is closed under multiplication. If g  HPH, then 
g   and g-  6 H a  = PH follows; PH is acone of H. 
Lemma 3.3. Let t  . Then PHXt = xt. 
Proof. Itis enough to prove this for t 6 [0, ), since t = k + t0, t0 6 [0, ) in the 
general case with x  in the center of G. 
If t = 0, then PHXt =  = xt. If t 6 (0, ), then for any j 6 J() there exists 
an n with wîxt  j by Lelnlna 3.1. Hen«e, xt = J() = Uwîxt ç PHxt, 
and the statelnent in the lemlna follows. 
The next result shows that F contains eleinents of a certain type. 
Lemma 3.4. For any integer m and ay  > 0 there ests an elemet xtu in F 
with "  U ad t  (m,  + ). 
Proof. Let h be the element www in F. Thon, by Lemma 3.3, we have 
I(x +) = x ° where fl = arg(). It follows that lh(x °) < l7(x ) = x - 
and that tkç (x °) < x -for any natural number N. 
 conclude that for the given integer , there exists a natural mmber N and 
an integer AI < m with 
For e the given rem number, there exists by Lelllma . and tlle contimlity of I a 
 with 0 <  < e and 
«, ([°, x)) ç (,x+) 
and hence 
(.) , ([x«+)) ç ((+«(+1+) 
follows for all k 
Bv Lemma 3.1 there exists a natural number n with 
çÇï,h(x °) [xM,x M+5) and h=h F. 
Hence, by (*) we obtain 
By another application of Lenlllla .1, there exists a lmtural llllnlber 2 with 
By repeating the last two steps m - (M + 1) tilnes, the statement of the lelnllla 
follows. 
The next result shows that the cones P are indeed exceptiolml. 
Proposition .. Let H  F be a sbgrop of  ad P =   H. The: 
a): P is an ezceptioal rak oe cone i H. 
defiees a isomorphism betwee () ad (P). rhe iwverse 4 ç is 
9iven b ç-(C) = C for C a divisorial P-ideal. 
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Pro@ We recall that AA(IF) is the group of divisorial IF-ideals in G (Definition 1.6) 
a.nd that AA(IF) = gr {Q} = gr (xN?) by Theorenls 1.9 and 3.2. 
If C is a divisorial PH-ideal in H. then C'IF is a IF-ideal in C by Lenmla 3.,-1. The 
divisorial closure C' of CIF is thereh)re equal to solne power of ,rrIF and C' = 
follows for some m in ]. çVe want fO prove that CI? ç3 H --: C and assulne that 
hPH D_ C for some h C H. Then hPHIF = hIF D CIF; hence hIF __D CIP. Therefore. 
hPH = hPçH D_ CIFç H. It follows tlmt C= C D_ CIFçH D_ C" and C" = CDH. 
This shows that Cbeing a divisorial PH-ideal ilnplies C = xIF ç3 H for some m. 
We want to show next that (z'IF-"-H) : :r''IF  H for anv u. Since 
is divisorial, we know that (a«'IF-'H) = xIF (3 H for SOlne m by the above 
al'glllll(qI. 
Bv Lemma 3. there exist elelnents x tul, xt2uz  F C H with 
N, u,,u2 G U a.nd ri,t2 G (Tf(tf - 1).Tf(t, - 1)+ ). 
It fillows that 

 rt'l PH = u.t UlIF f-? H D .r "- u2IF  H = x t u2Pu  x P  H. 
Hence, .r(n-l)P  H  (.rPH)  xP H. 
If «(-I)pH = (zPH), then this ideal would also be equal to .rtPH and 
 r t: uPH. This would ilnply xttl PHP = zP = ,rtuPH  = ztP, a contradiction 
that shows that (.r  H) = (zP H) for all n. This set of divisorial P-ideals 
does hot contain J(PH), does hot contain a completely prime ideal (Lelnm 3.3 
and 3.5) and no ideal of the folIn zJ(PH) ¢ J(PH), «  PH, is colnpletely prime 
in PH. This shows that PH has rank one and that «(PH) is infinite cyclic with 
QH = z  H as the positive generator of ,M(PH). Since J(PH) D QH. it follows 
from Theorem 1.9 that PH is a.n exceptional tank one cone in H. This proves all 
sta.telnents in the lemma.  

We consider now a condition that will guarantee that PH is exceptional of type 
(c). 

Proposition 3.6. Let H be a subgroup o.[G containing F with H (gr {x'} x U) = 
gr {x 'k} x U(PH) for some integer lr >_ O. Then the exceptional cone PH bas type 
(C). 

Pro@ It was shov«n in the previous proposition that PH is an exceptional cone 
with AA(PH) = gr{(xIF (3 H)}. To prove the statement in this proposition it must 
be shown that 7-l(PH) = gr {xPH}, sec Theorem 1.9. Hence, let gPH = PHg be 
a principal ideal in H (sec property e) iii Section 1.1). 
Then 9IF = PHgIF = IF9 ]P by Lenmm. 3.4 and gIF = IFg since IF has rank one. 
By Theoreln 3.2, c) it follows that g = .r" G H for some integer m and u c U 
and /7 = Xkr for u  U(PH) and solne integer n bv assumption. Therefore, 
gPH = xrrrpH and H(PH)= gr {xrtpH} = gr {Qk} follows for Q = xIFçH: PH 
is exceptional of type (Ck). [] 

Theorem 3.7. Let Hk = gr {u) 1, W2,.r rrk} be the subgroup ofG generated by F and 
the central element x  for an integer k >_ O. Then Pk = IF V Hk is an exceptional 
tank one cone in Hk of type (Ck). 

Pro@ It is sufficient to verifv the conditions iii Proposition 3.7 for H.. 
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Assulne that 
(*) "]Ç/r/';P'/ïl'/-'I /J-'ï2"/-' 2"'" u'ïnw n = .r=m*  Hk R (gr {z =} x U) 
for solne integers p. ui, Pi for i = 1 ..... , and u  U. b apply the mapping 
(Theorem 2.5b)) to both sides of thc above equation and obtain 
12Wl 1 î)...(2ïn 1 a 1) 
(**) (-1)P(o 1 )(2"1 
where u = (  
0 _1) with b, 0<aGN. 
Since the entries of the lnalrices al the left side are all integers, it follows that 
and a - are int.egers greater lhan Zel'O: hence a. = a - = 1. By a silnilm" argument 
it follows that b is an even inleger, b = 2s for some s in Z and u = (  ) = (  î ) = 
wf G r(F) follows. 
If (-I)P(-1)  = -1, then it follows from (**) that 
__(î) = (2ïl)(2 1 î)...(2n)( 1 
..î)( ) e (F), 
which is a contradiction, since the group r(F) fl'eely generated bv r(u,1) and 
(see the relnarks bcforc Lclmna a.3) does hot ('ontain a nonIrivial central elelnCnt. 
Therefore, (-1)  = (-1)  can 1,e can«clled in (**) and, using again Ihe fa.ct that 
b = 2tq if we ignore expon«nls lhaI could l»c zcro. Wilh u = wï ve tan rewrite 
= z tv . Il fifllows that tq = s, m = kp and u = 
the condition in Proposition 3.7 is satisfied and Theorem 3.8 follows. 

4. EXAMPLES OF EXCEPTIONAL ANK ONE CHAIN DOMAINS 
In this section we construct domains S associated with the exceptional cones 
Pk of type (Ck) as described in Theoreln 3.8. 
In Lemma 2.6(c) it was proved that TU is an Ore domain for any skew field T 
and the subgroup U of G.  denote by K the skew field of quotients of TU for 
a given skew field T; for exalnple, T = Q, the ratiolmls. Let lç{G} be the right 
K-vector space and Ieft T-vector space consisting of all series 
with ri < t < .... k  Iç. and supp () = {a "t' [ k ¢ 0} well ordered. 
XZ call snpp () the support of the series . If k ¢ 0. lhen v() = z ri 6 R is 
the norm of 7 and v(0) =  for 7 = 0. 
Let Q = End Iç{G}- be the elldolnorl)hisln ring of the K-veclor space Ix {G}ç. 
For q G Q and  G K{G} we write q[7] for the ilnage of  under q. The representa- 
tion V : G  Aut (R, 2) considered in Section 2.3 can be extended to a mapping 
V defined on Q with 
V(') = '(q[.']), V() =  
for q  Q, .t 6 R, and l  (R. OE)  (R, oe). It follows that 
ç+(a "t) E nlin{ç(z'). 
for any a, b  Q and z t  R. However, 1' is not equal to I o 1  in general. 
We recall a definition and a result given by Mathiak in [17]. 
Definition 4.1. Let D be a skew field and (F, ) a linearly ordered set. Then a 
lnapping 1" : D*  Aut (F, ) is called an M-valuatio if the following conditions 
hold: 
r 
MV1. I = I o l,g for any a,b  D*; 
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MV2. Va+b(h) >_ mill{l/(]), I/b(]))} for any a,b  D* with a + b -)/= 0 and h  C. 
If we add the syInbol oc for infinity to F and define V0(b) = oc and t(ec) = oc 
for all h  F, 0, a  D, then MV1 and MV2 will be valid for all eleInents a, b  D 
and all b  F U {et}. 
The next result follows almost directly froIn t.he previous definition; see also [16] 
nd [17]. 
Proposition 4.2. Let V  D*  Aut(F, _<) be an M-valuation for a skew field D 
and a linearly ordered set (F, <_) and let 5 be an element in F. 
Then the set Sh = {d  D I Vd(h) _> 5} is a total subring of D. Conversely, any 
total subring S in a skew fleld D can be obtained in this way for F = {aS I a  D* }, 
aS >_ bS if and only if aS c_ bS and I)(aS) = daS. The ring S coincides witb Sh 
forh=S  F. [] 
The space K{G} introduced above is also a left G-module if we define for g  G 
and 3' = '. xtlki  Iç{G} that 
g3` = :C t' (Ulkl) n t- xt; (U22) n t- xt (u3k3) -t- .   
where g. x t = xt',i for ui  U ç K, t' i  ll. It follows from Lemma 2.6(d) that 
t' < t < t < ... is also well ordered and hence g7  K{C}. The group ring TC 
can therefore be considered as a subring of Q. 
If A is any subring of Q. then we define 79[0. ,4] = A and 79[n + 1, A] as the 
subring of Q generated by 79[n, AI and ail inverses of elements of 79[n, ,4] in Q. The 
union 

U 79[n, A] = 79[A1 

n=0 
is called the rational closure of A in Q. Let I13 = 79[TG] be the rational closure of 
the group ring TG in Q. 
The following result can be round in [11] (see [10] also): 
Theorem 4.3. a) The rational closure I of TC in Q is a skew field. 
b) The mapping 1  restricted fo I* is an M-valuation of I* to Aut (R, _<). 
c) The rang S = {d  I13 I Vd(x°)  x°} is an ex«eptional tank one chain order 
in  of type ( C ) associated with the exceptional cone P in the group G.  
In order to construct skew fields that contain rank one exceptional chain orders 
of type (Ck) we consider the rational closure Dk = [THk] of the group ring TH 
for the group H = gr {u,,w2,x "} (see Theorem 3.8) in Q = End K{G}3. 
Since Dk ç  = [TG] C Q and  is a skew field by the above theorem, it 
follows that D is also a skew field and S = S  D is a total subring of D. 
It follows from Corollary 1.10 and Theorem 3.8 that S is an exceptional rank 
one chain domain of type (Ck) if the following theorem is proved: 
Theorem 4.4. The total subring Sa = S  Dr is associated with the cone Pk = 
P H. 
Before this theorem can be proved, we need the result in the following lemma. 
Lemma 4.5. Let   K{}. Then 
(,)  suppd[? l ç  I(supp?). 
dD gH 
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Pro@ Let I: be the right side in (,). Then iii order to prove (*) it is sufficient to 
prove 
(**) supp d[T] C_ }: 
for any 7  K{G} and any d  Dk = U [n, THk]. X will prove this in rive st.eps 
using induction on n for  the smallest index with d  [n, THk]. 
STEP 1. Assmne that d = xtu ¢ Hk, u ¢ U and that 
i<A 
0  ki  K for all ordinals i < A. 
Then d = xt:(uiki) for .rtu.r t" = xte,i  , ai ¢ U. Hence, suppd[] = 
i<A 
suppd= {x t{ l i< A} = {lh(xt') [ i < A} Ç {I(xt') l g¢ H.,i 
STEP . The inclusion (**) follows ilnlnediatelv for d ¢ T. 
STEP 3. Assulne that o, b  Dk with suppa[] U suppb[] ç I for any ¢ Iç{G}. 
Then supp (a+b)[] ç supp a[]U suppb[] ç }ç. Further, I(}') = } for g ¢ Hk 
and hence 
"PP( )[q ç ;[ = U ç("pp[q) ç U ç)= ,. 
gEHk gEHk 
STEP . It follows froln Stcps 1-3 that the statement (**) is true for ail d 
STEP 5. Assume that (**) is true for elements d  [, - 1, THk] for some n  1 
and all 7  K{G}. 
Let d = p-  [,,. THk] with p  [, - 1, THé.]. XX consider fl = d[7 ] and 
decompose fl into the sure fl = flo +fl with supp (flo) 
Th, v = p[Z] = p[£] + 
By the induction hypothesis, it follows that 
gHk gHk 
Hence, supp (pitié]) = supp (7-P[o]) ç supp TU supp (pi&)) ç ]. On the other 
hand, supp (pitié]) ç ]' since p e [n - l, THk]. If we assume that there exists 
 element h in supp (pitié]). then, on thc one hand, 
forsome gHk andsome h' supp(7) 

h = Ih' 
and on the other hand. 
 = V« (h") 

for solne g' E Ha. and solne h" E supp (fll)- 
This ilnplies h" = I@,}«9(h' )  Yv ç supp (/31) = 0, a contradiction that shows 
that supp (fl) is empty and supp (ff) = supp (flo) 
The ring [n, TH»] is generated by [n - 1,THk] and all elements p- for 
p  [n - 1, THk]{O}, and it now follows bv an application of Step 3 that (**) 
is true for ail elements in [n, THk] which completes the induction and proves the 
lemma (see also: [111). 
We now return to the proof of Theorem 4.4. 
Let d be a nonzero elenmnt in D. Since Dk ç D and S is associated with the 
cone , the element d can be decomposed as follows: 
d = xtm = qx t' with m,q  U(S), xt = xt' 
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(see Definition 1.2). It, follows from (*) iii Lelmlm 4.5 with 3' = x° that 
suppd[x °]C U I('r°)= U v(g}. 
gEHk gEH 
Hence, ,(d[x°]) = l(.r °) = I( o l(x °) = l:,(.r °) = ,c t since m  U(S), and 
hence .r t = v(g), g = ,rv, «  U for SOnle elelnent g  H. 
It follows that d = (.rtu)(u.-m) for xtu  H and u-m = (xtu)-d  Dk. 
Further, u-m  U- U(S)  Dt. = U(St.), since U Ç U(S) and S  Dt- = S. 
Applying the salue argulnent.s to the elenlent d - = x-t'q -1, we conclude that 
there exist.s an elelnellt f  Hk with 9  = x -t' w for SOlne w  U. Hence, we obtain 
a decolnposit Joli 
d-t t' lq-1) -lq-1 
= (x- u,)(u,- for « = (g,)-ld- e Dt. a lz(ç) = U(S). 
This proves the first hall of condition (ii) iii Definition 1.2. if we write d = 
(qu,)(u,-.rt'), q«,  U(Sk), w-ix t' = (9') -i  H. It renlains to prove the equality 
Px,P = pu-.r'p.. 
Let w-ix t' = ,t".t for sonw u   U and t  N. Since q" is associated with , 

it folh)ws that 

Therefore, 

IP.rt'lp = Ipxt"Ip. 

P.rtuP = Pt.xtu(Y c H) = Pt.xtY t Hk 
= lPxtlp çl Hk = IPxt'IP ç Hk = II%t"IP ç Hk 

[] 
Corollary 4.6. Tbe cbain domain Sk is exceptional of tank one and of type (Ck). 
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ON THE SPECTRAL SEQUENCE CONSTRUCTORS 
OF GUICHARDET AND STEFAN 

DONALD W. BARNES 

ABSTRACT. The concept of a spectral sequence constructor is generalised to 
Hopf Galois extensions. The spectral sequence constructions that are given 
by Guichardet for crossed product algebras are also generalised and shown to 
provide examples. It is shown that ail spectral sequence constructors for Hopf 
Galois extensions construct the saine spectral sequence. 

1. INTRODUCTION 
A. Guichardet [3] has given two constructiols for a spectral sequence with/ïT 'q : 
HP(G, Hq(], X)) and target the Hochschikl cohonlology H°(G x« B, X) of the 
crossed product algebra A = (7 x B. where (7 is a group, B is an algebra, c is a 
represeutation of (7 by automorphisns of B and the nmltiplication in A is given bv 
(g,b)(g',b') = (gg',a-,l(b)b'). 
Here, X is a left and right A-bimodule or, equivalently, a left module over A « = 
A ç-) A °p where A °p is the opposed algebra of A. These constructions are mmlogous 
to the Hochschild-Serre constructions for the spectral sequence of a group extension. 
He has asked if the methods of Barnes [1] can be used to show that they construct 
the smne spectral sequence. 
D. Stefal [5] has given a spectral sequence, based on the Grothendiek composite 
functor spectral sequence, for the cohomology of a Hopf Galois extension. 
The three contexts have some features in common. All have a "large" algebra A 
and the category .A of A-(bi)modules, a subalgebra B, a "small'" algebra (7 which 
plavs the role of a quotient, of .4 by B and the cat.egory C of Cnlodules and a 
category/7 in which the filtered cochain complexes are constructed. Ail have a left 
exact flnct.or ¢  .A --, C and a left exact functor ¢  C --, /7, raid the spectral 
sequences have as t.arget the right derived functors of the composite 0 = ¢ o ¢. 
Throughout this paper, ¢, ¢ and 0 will denote these functors. 
In Barnes [1], A is an auglnented algebra over a commutative ring .q and C is 
the quotient AffB of A bv a normal augmented subalgebra. A, B and A/lB are all 
assumed to be projective as .q-modules. The functor ¢ is given for the left A-module 
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X by 
eX-- XB= {.cE .\'lbx=e(b).r fo," all bE/3}. 
For a C-module }', P} = }-c and we have ppx = X A for any A-module X. 
In this context, p has a left adjoint j  C - al, and use is ruade of the counit 
7r = jp  al - al of the adjunction. Note that pj = id  C - C. \Ve shall refer te 
this as the HS context. 
In Guichardet's paper, A is the crossed product G x«/3. where/3 is an algebra 
over the field .ff, C is the group algebra JïG and the fimctor p is given for the left 
A«-module X by 
OX = X B = {.'r E X I b.r = :rb for all b E B} 
with the action of G given by g.r = (g, 1)x(g -1, 1) for g E G and x E X B. For the 
.¢G-module 
e"}" = }-G _ {y E }'IgY = Y for all g E G}. 
In this context, çb does net bave a lefl a(tj()int j. The assumption that .ff is a field 
can be weakened if in seine places we tel)lace "injective'" by "relativelv injective". 
\Ve require that . is a c(mmmtative ring and that B is )ï-projective. \Ve shall refer 
te this as the G context. 
In Stefan [5], C is a H()i)f algebra over the field ., A is a C-comodule algebra 
and B is the subalgebra of coinvariants. Thus we have an algebra morphisln XA " 
A -- A  C making 4 a right C-comodule, and 
B---Aç°c= {«,E A I,X4()---® 
Stefan refers te this situation as "'the extension ,4/13"'. It is assumed te be C-Galois. 
which we explain in section 2 below. As in the G context, for the A-bimodule X, 
we set px = X B. This is ruade into a right C-module using the action defined 
by Stefan [5, Proposition 2.3] and explained following Lemma 2.1 below. For the 
C-module Y, we put ¢;'1" = }-c. Again, we veaken the &sumption that .ff is a field. 
\Ve require that . is a commutative ring and that A. B and C are )ï-projective. We 
refer te this as the S context. It generalises the G context since the crossed product 
algebra A = G × « B becomes a C-comodule algebra if we set x.4 (9, b) = (9, b) ® 9 
for 9 E G and b E/3. If in the HS (Hochschild-Serre) context, .4 is a Hopf algebra. 
we mav regard it as a C-comodule algebra with the comodule strncture given bv the 
conmltiplication of A followed by the natmal homomorphism .4 ® 4 -- .4 ® AffB. 
A left A-module X mav be regarded as a 1)imodule by setting xa = e()x for a E _4 
and x E X, where e is the augmentation. This does net change çX = X , new 
defined as {x E X ]bx = xb for all b E B}, ner does it change the Rqcp(X), although 
it does change the injective modules used for their calculation. 

2. PRELIMINARIES 

We follow the notation for comodule algebras used in Schneider [4], with the 
exception that we denote the Hopf algebra by C, reserving the symbol H for coho- 
mology. Thus we have the comodule structure map, AA : A --, A ® C and express 
the image of an element a E A bv A.«(a) =  a0 ® al. The comultiplication 
Ac : C  C0,C is written Ac(c) = yc ®c2. The augmentation of C is denoted 
by e and the antipode by S. The canonical map ean : A ® A ---* A ® C is defined 
by can(a ®B a') =  aao ® al . That A/B is a Hopf Galois extension means that 
ean is invertible, which we always assume. Thus for c E C. there exist elements 
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ri (c), li (c) E A, hot mfi<luely detern,ined, such that tan- 1(1 ®c) =  ri (c) ®B li (c), 
which is unique. \Ve shall need thc identities proved in Schneider [4. Ptemark 3.4(2)]. 
Throughout his paper, Schneider assumes .ù, to be a field, hut his proof of the iden- 
tities makes no use of that assumption. For the convenience of the reader, we list 
the identities here. 

Lemnla 2.1. For all a  .4. b  t3 atd c, c'  C, t]e followin 9 identifies ]old: 
(a)  bri(«) ®B li(«) =  ri(c) B li(«)b, 
(b) 
()  («)h(«) = («). 
(d) r,(«).i(«)oli(«) = ,',(c)li(c)c2. 
() 
() 
Following Stefan [. Proposition 2.], we use the above relations to d«fine a right 
Cmodule structm-e on X 
we put .r- c =  ri(c)xli(c). This is well defined since  ri(c) B l(c) is a well- 
defined elelnent of A @B A alld b.r = .rb for ail b  B. Froln 2.1(a), it follows that 
x. c  X B. If a left açtion of Ç on X B iS preferred, o11(' lllaV be defined by setting 
c- = x-(Se). The sertion of LOlnlna 2.2 below holds for this lef action provided 
that the antipode S is bijective. 
Lemma 2.2. For an Aç-modle X . ( X e) c = X A. 
Proof. For z  X A and c  C, we bave 
by 2.1(c). Thus z  (XB) c. Conversely. if x  (XB) c, then by 2.1(b), 
for all 
For the crossed product algebra A = G x B, the canonical lnap is given bv 
b 
Cl((,)('.v'))=(, )(,') =('.@()'). 
In particular, can((9 -1, 1)B (. 1)) = (1.1): so we can take r(9) = (-. 1) and 
l(9) = (9.1)- The right a.«tion of C G on X B be«onmsz-9 = (9-1.1)z(9.1) 
for z  X and 9  G. Converting this to a left action gives   z = (, 1)z(9 -1. 1). 
which is the action used in the Guichardet paper [3]. 
Note that, in the G context,..4 « = A  A °p is free as a right B«-module. In 
the S context, we assunle that A is fiat as left and right B-module. If tlmn follows 
that A « is fiat  right B«-lnodule. In the HS context, we assume that A is at least 
projective as right B-module. At some points in [1], the stronger assunlption that 
the modle quotient A/B is projective as right B-module is use& In all the contexts, 
by Barnes [1. Lelnlna 1.4.3], everv il0ective left A- or 4C-module is injective as B- 
or B«-module. Further, every il0ective of M or C is injective as .q-module. 
For any .q-module X, the A-module coinduced froln X is the module X* = 
Hom.(A, X) wit.h the action (af)(a') = f(a'a) for f  X* and a,a'  A. (For the 
coinduced right lnodule, the action is given by (fa)(a') = f(aa').) If X is itself a 
left A-module, then the nmp «  X  X* defined by («.r)(a) = ax for a" q X and 
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a E A is a .R-si»lit A-module monomorphism. X* is a relatively injective left A- 
module. (See for example, Barnes [1, Lenuna II.2.4, p. 20].) A module is relatively 
injective if and only if it is a direct sunmmnd of a coinduced module. By Barnes 
[1, Lenmm. II.3.9, p, 28]. if A is right B-projective, then everv relatively injective 
left A-module is relativelv injective  left B-module. Thus in the G context, every 
relatively injective A«-module is relatively injective as B«-module. In the S context, 
to get this conclusion, we nmst strengthen Stefan's assumption that A is left and 
right B-fiat t.o A being left and right B-projective, although as t.he next lemma 
shows, this strengthening is mmecessary if, as in [5], if is sumed that  is a field, 
since then, every module is -injective. 
Lemma 2.3. If X is -injeclive, then the coinduced module X* is injecive. 
Proo: Let i  I"  IV be a monomorphism and a  I"  X* a honomorphism of 
A-modules. 
i 
X* 
We want to construct a. homomorphism fl  II"  X* such that i = a. For 
v  I , we bave (tv  Hom.(A, X); so for a  A. we lmve (v)(a)  X.  m- 
regard a as afimction A x 1"  X, writing a(a,v) for (av)(a). For b  A, we 
have a(bv) = b(av). So (a(bv))(a) = (av)(b); that is, (ab, v) = a(a, bv), and in 
particular (a. v) = a(1, av). Putting (v) = (at,)(1) creates the diagram 
of -modules. Since X is -injective» there exists a -homomorphism   I1  X 
such that Bi = . Define " II"  X* by (w)(a) = B(aw). Then for b e A. we 
have 
/(b)() = (b) = (,,,)(b) = ((,,,))(); 
so/3 is an A-module homomorphisn.  have 
(,,)() = $((i,,)) = ((,)) = (,,) = (,)() 
and fil = a. 
We have defined a right C-module tion on X u for any leh A«-module X. 
need another description of that action in the case where X is a coinduced module. 
Lemma 2.4. Let V be a R-module and let I'* = Hom(A , V) be the coinduced A - 
module. Then V *B is isomorphic to the coinduced C-module Hom(C. Hom(A, 1")). 
Proof. The tion of a @ a'  A « on f  V* is given by 
(( e ')I)( e )= I(( e )( e '))= 
for x, y  A. Now 
Y*= {f  Y* I (be )I = ( e b)I for fi   } 
= {f  V* f(xb@y) = f(a" @ bç) for ail b B and x,y in A}. 
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Thus 1"* B can be identified with Hom(A®u A, V), and so, using the canoifical map, 
with Hoin(A ¢_ C, 1;) = Hom(C, Hom(A, V)). We calculate the right C-modnle 
action Oll Hom(C, Hom(A, V)) induced by these identifications. For f  V *B and 
c G C, froln the A%a.çtion on 1", we get fc = 7,(ri(c) C' li(c))f. Thus 
(fc)(x ® g) =  f(xr,(c) ® li(c)g). 
For f  Hom(C, Hom(A, V)) we have the corresponding f'  Hom(A ® C, 1") 
given by f'(o ®c)= f(c)(a) and f"  Hom(A ®8 A. V) givell by 
f"(x ®B U)= f'(ean(x ®B y))----  f'(xgo ® U)- 

Thus, 

f(c)(a) = f'(o®c)= Zf"(ori(c)®Bli(c)). 
So for c'  C, fd is given bv 
(fc')(«)(a) = Z(f"«')(ari(«) ®B li(c)) 
=  f"(""(«)"(ç') ®- 
= f"(ari(c'c)@.li(c'c)) by 2.1(f) 
= f(d«)(a). 

Thus the action (fc')(c) = f(c'c) is that of the coinduced right C-module. [] 

The next result strengthens Stefan [5. Proposition 3.2]. The corresponding result 
in the HS context follows easily froln the fact that every AffB-module is an .4- 
module and that QB is a submodule of Q. 

Lemma 2.5 . Let Q be a relatively injective A¢-module. Then QB is a relatively 
injective C-module. If Q is injective, the QB is injective. 

Pro@ Q is a direct smnmand of some coinduced module I "* = Hom.(A , 1"). So to 
prove Q8 relatively injective, it is sufficient to show that Hom.(A «, V) B is relatively 
injective. But Hom(C. Hom(A. 1")) is relatively injective; so by Lemma 2.4, V *B is 
relatively injective. Thus QB is relatively injective. If Q is injective, we nav take 
V = Q. Since Q is .¢binjective, by Lemnm 2.3, Holn(A. Q) is also .R-injective and 
Hom(C, Hom(A, Q)) is an injective C-module. Thus QB is injective. [] 

3. SPECTRAL SEQUENCE CONSTRUCTORS GENERALISED 

In Barnes [1, Chapter III] in the HS context, a spectral sequence constructor for 
(¢, ) was defined to be a functor F from ,4 to filtered cochain COlnplexes in T) such 
that 

(1) F is exact (in every filtration). 
(2) F is acyclic on injectives; that is, if Q is injective, then H'(FQ) = 0 for 
n > 0 and for all p. Hq(°FPQ/°Fp+IQ) = 0 for q > 0. 
(3) E°F is exact on C. 
(4) The inclusion i" X B --, X induces isomorphislns Eï°F(X B) --, Eï°F(X) 
for all p and ail X  .4. 
(5) H°F is naturally isomorphic to 
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Here, using the fact that A//B-nlodules are A-modules, that is, using the adjoint 
j to b, we c'onstruc't a flmc'tor F : '" 
1  fi'Olll Cto cochain complexes in . This 
cannot be done in the G or S contexts. So we nmst include the cochain complex 
flmctor as part of the structure in our definition of a constructor. If F is a filtered 
cochain complex, we denote the colnponent of total degree n bv F, the pth filtration 
bv F', the submodule of filtration degree p and C'olnptementary degree q by F pq 
and use similar notation fbr the t.erlns of its spectral sequence E(F). e always 
assmne that "F ° : "F and that 'F +1 : 0. The following definition generalises 
the Olle quoted above. 
Definition 3.1. A spectral seqllcnce constructor for the pair (, ) is a quadruple 
(F. F. /. 7), wherc F is a functor from A to filtered cochain complexes in , F is a 
fimctor ffoto C to cochail c'omplexes iii , I I is a natural isonlorphisln l ("' F)  F& 
and 7 is a natural isomorphism H°(F)  ç such tha.t 
(1) F is exact (in everv filtration). 
(2) F is acyclic on injcctivcs: that is, if Q is injeçtive, then H"(FQ) = 0 for 
n > () a.nd tr ail p, Hq('FPQ/"FV+Q) = () for q > 0. 
(3) I" is exact an(t a.«y«lic on inje«(ives. 
From (1), il follows that "FV'F + is exact for all p, r. From 7 being a natural 
isomorphism and (3), it follows that we have a torique fanfilv of natural isomor- 
phisms 7 p Ht'P + RPç, the right d('rived flm«tors of , «ommuting with colnmct- 
ing homomorphisms, and with 70 = 7- k, denote this family bv 7. Furthermore, 
H'F = E°°F._ = E°F = H°Eï)F = H°q-iF, 

and SO 

"H°(71) H°F = 7 = O: 
that is, "7H°(7/) is a lmtural isonorphism flOlll H°F to 0. It follows that H  F = RO 
for all n. 
X now prove thc results corresponding to Barnes [1. Theorem III.2.3. p. 42] 
and the lelnm leading up to that theorem, begimfing with the analogue of [1. 
Theorem III.l.5, p. 38]. 
Lemma 3.2. Let (F, F,I.) be a spectral seq,ence con.tructof There exists a 
unique famil of natw,'al t','ansformations ,1 pq " Eï ( F)  F" ( H q¢) coin m uting with 
connecti9 homomorphi.s'ms and with I1 pO = I1 p. Ile ]Pq are atural isomorphisms. 
.sati.sg I]P+l"qdï q : (-1)qgl] pq. where d is the differential 4 F. and induce 
isomophisms 
(,'")- () 

and 

HP(q°q) - Eq(F) --> (R'b)(Rq¢). 

Pro@ F is exact by assumption. So {F"(Rq¢)[q = 0.1 .... } is a connected sequence 
of fun«tors, as is {Eï(F) = Hq(F"/F"+)]q = 0.1 .... }. Both vanish for q > 0 
on injectives. By dimension-shifting, it foltows that there exists a torique family 
of natural transformations 1 "q - Eï(F)  ['P(/ïqçD) extending the given tra.nsfor- 
mation qP  Eï°(F) ---, FP¢. Since /p° is a natural isomorphism, all the t' are 
natural isomorphisms. The argument of [1, pp. 38. 39] applies unchanged to give 
the result. [] 
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Definition 3.3. A natm'al transibrmation (  (F. F. q, 7) -- (F'. F'. q'7') of con- 
structors is a pair F " F -- F  and çFÇ -- I , Of natural transformations such that 

the diagralns 

COlllnlllt (L 

VVe shall onfit thc subscripts from çF and çr- Out next lemma is easier than 
[1, II.2.2] in that the transformation ç  F -- F' is given instead of having to be 
constructed. 

Lennna 3.4. 
constructors. 

Let   (F, F, '1, 7) -- (F', F'. 1', 7') be a natural tra,fformatton of 

Then the diagram 
:ït ( r )  
-- FP Rqc 
Eïq(F,) '/ , F'PRqdp 

commutes for all p and q. 

Pro@ çH°ç is a natural transformation of connected Se(lUellCeS of fimctors. Since 
in dimension q = 0, we have R°¢ = I//P0(I/P0) -1, tV dimension-shifting, we have 
çq¢ = ?]tpqÇ(?]pq)-i for ail q.  
Theoreln 3.5. Let ç  (F. F. q, 7)  ( F. F, q', 7 ) be a natural traufformotio, of 
constructors. Then ( induces a natural isotorphism. çE " E(F)  E(F ) of their 
spectral sequences, that is, (.q  Eq(F)  Eq(F ) is a natural isomorphism for 
all r  2 and all p. q. 
Proqf. Bv assumption,  " H°(r)  , and 7' " H°(F')   are natural isomor- 
phisms, and 'H°(ç) = . Therefore H°(ç) = (')- is a natural isomorphism. Bv 
dimension-shifting, it follows that HP() " H  H  is a natm'al isomorphisln 

for ail p. The diagram 

is, up to sign, a commutative diagram of cochain COlnplexes. (If q is odd. q and 
q' anticommute with the differentials.) Taking H p of this, ve get the commutative 

diagram 

in which '1, '1' and HP(ç) are natural isomorphisms. It follows that çq is a natural 
isomorphism and so, that ,,cm is a narreal isomorphism for all r _> _.° [] 
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GI. ICHARDET S FIRST ('ONSTRUCTOR 
Foi" the Ae-lnodule X, Guichardet defines the double colnI)lex 
içpq(X ) = (2r(av+I,Hom(®V+IA«,X))B)a 
with apI)ropriately defined differential, where .T-(b, V) denotes the set of furie- 
tions rioto the set U to the set I . Following Guichardet [3], we set I'(X) = 
Hom(® '+IA , X) with differential 
! 
d,f(oo ® o'o ..... o,+, ) (/ri+l) 
! 
= a0,f(o ®(4 .... ,o+  
( ' , , 
+ -- 1)i+,f(ao ® o' o ..... aiOi+l @ ai+lai,.,  an+l @ an+ 1 
0 
which gives a relatively injective resolmion l'(X) of X in A. Also following 
GuMmrdet, we put Pn = ®'+I-R(,' with action g(9o ®... ® .On) = .O.Oo ® ... ® .O.On 
and set d(go ®... {" g,,) ' ' = 
= Yi=0(-1) (9o®...'i. -®g,,) and e(90) 1. This makes 
Po a ff'ce resolution of . iii C. We then have 
lçoq(X) = tlom.G(Po, Iq(X)B). 
Expressed iii this way, it is the Grothcndiek repeated (relatively) injective resolution 
construction foi the spectral sequcnce of a composite flmctor discussed in Barnes 
[1, Chapter VIII, with Hom.(Po, ) used as the relatively injective resolution fimctor 
on C. For any relatively injective resolution flmctor I" and any projective resolution 
P.. setting KPq(X) = Hom.c;(Pp, Iq(X)B) gives a constructor (K.F, 0,7) with 
F = Hom.c;(Pv, ), r = id and 7 = id. The spectral sequence constructed is 
independent (from the E2-1evel onward) of the choice of I ° and of P.. 
This constru«tor mav also be regarded as a.n adaptation of the Cartan and Eilen- 
berg pair of resolutions constructor dis«ussed in Barnes [1, Chapter VI]. Since 6"7- 
modules are hot A«-modules, we cmmot use Homa, (Pv, Iq) as in the HS context. 
but use instead Homc(Pp, (Iq) B) which, iii the HS context, is essentially the smne, 
Stefan in [5] establishes the conditions foi" the Grothendiek composite flmctor 
spectral sequence. To obtain a. spectral sequence constructor, we bave merelv to 
assign ffmctorially the resolutions used iii the construction. If we assume that A is 
left and right B-projective or if we assume that Jï is a field, then we can use the I n 
defined as above and an,v right C-module projective lesolution P. of 

5. GUICHARDET'S SECOND CONSTRUCTOR 
For his second construction. Guichardet defines a filtration on the normalised 
standard COlnplex °N(A. X) where N(A, X) is the subspace of Hom.n(®A. X) of 
functions f for which f(a ..... a) = 0 if anv of the ai is in .1, and 
df(al,. .. ,an+l) 
Yt 
= al,f(a2 ..... an+l) n u -(--1)i,f(al ..... aiai+l ..... an+l) 
i=1 
+ (- 1)'+1 f(al ..... (-In)On+ 1 . 
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The filtration on this colnplex is given by defining n_N° 
tobe the subset of those fllnctions f satisfying 
f(al .... , aq, glbt,..., gpb v) 
--i (bl)O, L1...gp(b2 ) o,;pi(bp_i)bp. 
= f(o1 ..... aq, gl,''', gp)OZg2g3...g p "'" 
Gnichardet t.akes for F the normalised standard complex and constructs a natnral 
transformation from E°(N) t.o ['b which, in [3, Lennne 3.11], he shows is a nat- 
ural isomorphism. The conditions for a spectral sequence constructor are clearly 
satisfied. The purpose of this section is to gcneralise this to the S context. 
To nse the nornlalised standard COlnplex in the S context, we must ilnpose a 
further condition on the algebra A. The theory of the norlnalised standard coin- 
plex (Cartan and Eilenberg [2. p. 176]) reqnires that the quotient A = 4/.ql be 
projective as .-module. We assnlne this in this section. Equivalently, we assume 
that there exists a .-linear map  " A --, .q snch that (ki) = k. This condition 
alwa.vs holds if . is a field or if, as in the HS context. A is ail anglnented algebra. 
An equivalent definition of nNP, also given by Guichardet, is memfingful iii the 
S context. So we use it hele but with sides revel'sed because of onr nse of right 
COlnodnle algebras and right C-nlodnles. Fol" p >_ 1, we define A "p to be the subset 
of 'N(A, X) of those filnetiolls satisfying 
(5.1) f(bal,. ..,a_i,an) = bf(al,....a) 
and 
(5.2) f(al,...,ai-ib, ai,...,an) =f(al,....ai-l,bOi,...,an) lori=2 ..... p 
for all a ..... a  .4 and b  B. 
The normaliscd standard complex °N(C, )') for a right Cmodule t" is that ob- 
tained by treating )" as a bimodule with left action c- g = (c)9. Thus, N(C is the subspace of Hom.(®*C )') of functions f for which f(c ..... c) = 0 if any 
of the ci is in .fil. with the differential 
n 
df(cl,.. . ,c+) = e(c)f(c2 ..... c+_)+ Z(-1)if(«l,...,«iCi+l,... ,c,+) 
i=1 
+ (--l)n+lf(cl ..... Ch)Ch+l. 
We put T pq = PN(C, q]v(13, X)) and write N pq for P+qN p. Note that, although 
N(B. X) is hot, in general, a Cmodule. this does define .-lnodules T pq. 
For f  Nvq(A, X), we put 
qPq(f)(Cl ..... Cp)(bl,..., bq) 
= Zri(cP)...ri(c)f(li(c),...,l,(cp),b,...,bq). 
That O2Pq  N pq -- T pq is a well-defined .-linear map follows from the next lelnlna. 
We shorten the notation by writing (ï for a string al,.  , Oq of elements of A of any 
length. Ve flirther shorten notation by onfitting unnecesary subscripts from the 
(c),h(c). 
Lemma 5.3. If   NPq(A. X), then for j = 1 .... ,p. and ail   A. b  t3 and 
C, C1,...,C p  C 
(1)  r(cj) . . . r(cl)f(l(Cl), .... l(cj),) is independent of the choice of the r(cj) 
and l(c). 
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(2) '. b'r(cj)'r(cj_l)...'r(cl)f(l(Cl) ..... 
=  r(cj)...r(cl)f(l(c) ..... l(cj)b. 6). 
(3) y r(c)'r(c)...r(c)f(l(ci) ..... l(cy)l(c).) 
-  r(cjc)...r(cl)f(l(c) ..... l(cjc).ç). 
Proof. For o. o'  ,4. we put 
gj(a @ a') =  a,'(cj_)...r(q)f(l(c) ..... l(cj_),a',g). 
Bv the condition (5.1), g is well defined. Thus (1) holds forj = 1. Also. by putting 
, -}B a'=  br(Cl) eB /(Cl) =  r(Cl) @B 
bv Lelmua 2.1(a). we see that (2) holds folj = 1.  use induction over j. 
For I <jp, wehave 
=  or(cj_)...r(Cl)f(l(c,) ..... l(c_),bo'.6) 
1)v the induction hyl)othesis that (2) holds for j - 1 and condition (5.2). Thus 
gj is well defined. Putting « '}B o' = r(cj)$ l(c) gives the assertion (1). 
Putting a OEB a' = b,'(c)@B I(cj) and using Lemlna 2.1(a) gives (2). Putting 
a 'B a'=  r(c)r(cj)@B l(cj)l(c) and using Lelmna 2.1(0 gives (3). 
Lemma 5.4.  defines a nalural cochain map o'" " E'"  T v'. 
Pw@ If f  nNp+I, then 
¢çq(f)(c,....cp)(b ..... bq) =  r(cp) ...r(cl)f(l(Cl) ..... l(%)b, 1,b ..... bq) 
= O. 
Since "ç0 = NP/"N p+, Pq defines a .q-linea.r lnap ¢q  Eg q  T pq. Consider 
the eXplessiol for (daf)(l(q) .... , l(cp), b ..... bq+). For those terlns in which 
the string of/(ci)'s is reduced in length, we get bi in the pth place: so those terres 
are 0. Thus. 
( P'q+ l d 4f )(c1 ..... Cp )( bl ..... bq+l) 
= (-1F ,'(%)...r(q)f(l(c),....l(%)b ..... b+) 
+(-l:+>(«)...y( .... (Vb+) .... ) 
+ (-:++ («)---/(I(«,) ..... )+ 
= (-1)PdB((Puf)(q,...,cp))(bl ..... bq+) 
by applying Lelnlna 5.3(2) to the first terln. The result follows, the naturalitv being 
obvious. 
 are trying to construct a spectral sequence coustructor using Aç4 = N(A. ) 
with the Guichardet filtration as the filtered complex functor. Clearly, we tan set 
F = N(C, ) and 7 = id  H°(C, )  .  still need a natural isolnorphisln 
q  E°(A4)  Y. Applyilg H q to the natural cochain map ¢" gives a natural 
map qPq  Eï  PN(C. Hq(B. )) = FP(Hq(B. )).  nnlst fil'St show that 
is al isomorphisln of cochain complexes. 
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Lemma 5.5. ,l "q " Eï(X) --, F°X B iS a map of cochain complexes. 
Pwof. An elelnelt of Eï°(.¥) is rel,l'eselte(l 1,y a fllll('tiOll f  PNP(A. X) such that 
f  P+INp+I. For f  PN p, every terln t in df satisfies (5.1) and (5.2) for ail i except 
the terre t(a ..... ap+) = f(al,..., ap)ap+l [or whi«h (5.2) lnay rail for i = p + 1. 
Thus the l'eqlfirPlnelit that dr  p+INp+I ilnpOses the OlW extra conditiol that 
f(al ..... apb)ap+ = f(a .... ,ap)bap+, that is, f(a ..... apb) = f(a ..... ap)b. 
For su«h an f. we have, wriling  ri)l" c 1 ..... Cp+l, 
(P+'°dAf)( =  r(cp+)...r(c)«f(l(Cl) ..... l(Cp+)) 
=  r(«+l).., r(«l)t(«)f(t(«) ..... 
+ (-)'r(«»+)... ,'(«l)f(tt«) ..... («)t(«+l) ..... 
+ 
By Lemma 2.I(c), 
= (Cl) r(%+l)...'(«)f(l(ç) ..... 
By Lcnmm 5.3(3), 
(-1)(ç+l)... («)/(t(«),..., («,)t(ç+l) ..... (ç+l)) 
= (-)',.(«+)...,.(«.«+)... ,-(ç)f(t(«) ..... (ç«+) ..... 

ï r(«p+l).., r(«)ï(l(«),.. 
= (Zr(«p)...r(«l)f(l(«l) ..... l(«p))) "Op+l. 
Thus kP+LOdAf = dcq2P°f and the result follows. 
For g  PN(C, xB), we define dpg  P/'(A..\-) by 
((I)g)(c/1 ..... c/p) = Z G10"" 0 1 1 
 
o 
wl'it.ilig the comodule structure indices as supel'scripts, 
Lemma 5.6. Oi, g e PNP(A. X) and d_4(Oi, g) 
Pro@ For b  B, we have 
sincc AA(bal)= OE haï ® a l. Thus condition (5.1)is satisfied. Also, 
( Opg) (a ..... aih, ai+l,..., 
__ Z(lï.. 0 0 0 1 llp) 
-- a i bai+ l .. 
 ap9(a,.... 
= (dPg) (al ..... 

[] 
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Thus (5.2) is satisfied for ail i and g E I)NI)(A,X). Since g(a ..... @)  X , 
(9) (al .... , avb) 
.... %b(1,...,) 
= aï.. 0 1 aç)b ...... ap)b 
 ap9(al,..., 9(al 
and it follows that dag  P+INp+I 
Lemnla 5.7. For 9  PN(Ç, xB), PO 
Pro@ For any .-linear flmction t  C  X, setting u(a  a'  c) = aa't(c) for 
a. a'  A and c  CdefilleS a .-linear function a  A @B A @ C  . Bv Lemma 
2.(a), 
 r(«)(c)°tq(«) ) = .,,( r(c)/(«)0  (c)) 
= 
=  F(cl)I(cl)t(c2) 
=  ¢(cl)t(c 2) by Lemma 2.1(c) 
= t(c). 
Using this with t(c) =  l(c2)°.., l(%)°g(c,/(c2),..., l(cp)l), we have 
(çP0)(C1,... Cp) = '(Cp)...r(Cl)()(/(Cl) .... 
= r(Cp)...r(Cl)I(Cl)O...l(ep)°(l(Cl) 1 .... ,l(cp) 1) 
=  r(%)...r(c2)l(c2) ° . ..l(%)°g(c,l(c2)  ..... /(%)1). 
Repeating this argument gives the result. 
Lenlma 5.8. If f  PNV(A,A ") and dal  P+I'p+I(A,ç). then po f = f. 
Pro4 Setting u(a  a') =  ar(@_l)...(l)f((l) ..... (-1).') defines a 
.-linear function u" A @B A  X by Lenlma 5.3(b) and condition 5.2.  have 

1Repeating this argument 

ai)r(ai))...O 1 ,-(l)f«(l),....,(1)) 
0 0 1 l(alp)) 
ai)-lZ(apr(ai)) @B 
@-lU(1 ®B ai)) by Lenlma 2.1(b) 

E aï " " " ai)--lO ir(a ..... ) (I)(,(I),.. ,/(apl--1), ai))" 
gives t.he result. [] 
Corollary 5.9. The ri I)° are isomorphisms. 
Pro@ Eï ° is the set of f  I)N(A,X) with dal  I)+INI)+I (A, x), and i11)° is the 
restriction i)01Eï ° --' I)N(C, XB). By Lemma 5.7, it is surjective and, bv Lemma 
5.8, it is injective. [] 
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Theorenl 5.10. Suppose  = A/fil is projective as .ff-module a'nd that A/B is 
projective as lift B-module. Tben (NA,F,I, ) is a spectral sequence coustru«tor 
Pro@  have to show that the couditions (1), (2), (3) of Definition 3.1 are sat- 
isfied. Since 
"NA (X) = Holn(@(.), X) = Hom(.. H,,ln(@ - .4, X)) 
and . is .-projective, N is an exact flmctor. A flmctiou f ¢ n satisfies the 
further couditiou f(ba .... ,a) = bf(a,...,a) for all b ¢ B. In particular, 
f(a,...,a.) = 0 ifa ¢ B. Thus 
nNÀ (X) = Holn(n A, HomB(A/, X)). 
Thus nN is an exact flmctor. Similarly, 
(X) = nom(.4, no,,,(A/, X)) 
and by iuduction over p, N is exact. Thus condition (1) holds. 
Let Q be an i0ective A«-mo(hfle. Then H([4Q) = 0 for n > 0 by the usual 
theory of the nornmlised standard complex. X have to show that 
u ¢',v O'-x[ +  O) = 0 

forq>0. But 

Hç(°NPAQ/°N+Q ) = EïA'A(Q)  PN(C, Hç(B,Q)) 
by Corollary 5.9. But Hç(B, Q) = 0 for q > 0 since Q is injective as B««nodule. 
Thus conditiou (2) holds. 
Sin«e F(}')= N(C, }'), «oudition (3) holds. [] 
111 the discussion of the filtered nornlalised complex in Barnes [1. Chapter IV], 
the corresponding extra assumption that A/B be projective as right B-module was 
needed. If in the HS context, .4 is a Hopf algebra, then it cau be regarded as a 
A//B-COnlodule algebra. The Guichardet filtration is not the saine as that given by 
Hochschild and Serre, but by the result of the next section, the two filtratiollS give 
the sanie spectral sequence. 

t3. UNIQUENESS OF THE SPECTRAL SEQUENCE 
As in BarlleS [1, Chapter X], we construct for each cardiual a, a coiffe fmlctor 
which, restricted to the subcategory .A of objects of cardinalitv less than a, is a 
spectral sequence constructor. (This use of the subcategory .A is necessary because 
a coiffe functor with injective lnodel /I aud injective basis (M, U) only has the 
desired properties with respect to lnodules embeddable in M.) From the existence 
of this cofree flmctor, we deduce as in [1, Chapter X], that all spectral sequence 
constructors construct the saine spectral sequence. We need one technical lemlna 
to get around the diflïculty caused by C-modules not being A%lnodules. For this. 
we again need the assumption that ri is q-projective, that is, that there exists a 
.ff-module homomorphism e : A -- q with e(1) = 1. 
Lemma 6.1. For every C-module Y, tbere exists an ijective A«-module Q such 
that Y can be embedded in Q. 



2768 DONALD ,V. BAF{NES 

Pro@ We first lnake Z = Hom(A, Y) a C-module l»y defining (f. c)(a) = f(a)c for 
c  C. a  A and f  Hom(A. Y). X construct an embedding i : }" + Z bv setting 
(ig)(a) = e(a)g for g  Y and a  A. S« defined, i is a C-module homomorphism, 
because 
(i(c))(«,) = ()9« = («(«,))c = (@(,,)« = ((+)  «)() 
tbr g E G, a E A and f E Hom(A. }'). It is clearly injective. 
Next, we use the standard embedding of Z in the coimhlced C-module Il = 
Hom.(C, Z), defining « : Z + II l»v setting a(z)(c) = zc for z E Z and c 
now ha.ve an embedding of Y in Ho»re(C, Hom(A. }')). By Lemma 2.4. we bave an 
embedding of }" in X B where X is the coinduced A:-module Hom(A «, }'). Taking 
any embedding of X in an injcctive A'-module Q, we get an embedding of 
 use the thcory of cofree filnctors develol»ed in [1. Chapter X]. Out spectral 
scquence constructors consist of two ftmctors and txvo natural transformations in- 
stea<l of the single flmctor used in the HS context. To accommodate this. we shall 
sav that the pair (F, F) of fimctors. F defined on A and F defined on C. is simple 

«ofrec on the 1)asis (5I. r, V) if F is cofree on (M.U) and V is cofree on ((M). V). 
Theorem 6.2. Let C be ( Hop.f «l.q«bra over .q and let A be a rigbt C-comodule 

algebra with B = 4 `c. Suppose ,4. B and C are .-projective atd that A/B is C- 
Galois. Let ,,4 be the categorg of left A"-modules, C tbe category of right C-modules 
atd let 73 be the categorg of .-moddes. Let  : ,A ---+ C atd ¢ : C -+ 73 be the 
flmctors defined bg ¢(X) = X B and '(}') = I "c for X  ,,4 atd Y  C. Suppose 
.4/.ql is .q-projective ad tbat .4 is both lefl attd right B-fiat. Then. for atg cardinal 
n. there exists a simple cqfree pair (T. F) with ijective model M ad injective basis 
(M. U, I'). and atural transformatiots 1.7 such that on the subcategory +-l of 
objects of gt of cardimlit.q less tha . (T. F, q,"f) is a spectral sequetce cotstructor 
for (¢, ). 

P.v@ By replacing  bv a suitable larger limit cardinal, we nlay suppose that everv 
object in A¢, bas an injective resohltion in A, and that every object of C likewise 
has an injective resohltion in C. There exists an injective module X in A such 
that every object of A can he embedded in X. Likewise, there exists an injective 
module I" in C StlC}l that e»vry object of C can be embedded in Y. Bv Lemma 
6.1. there exists an injective lnodule Q in M such that }" can be embedded in (). 
Putting Al = X @ Q. we obtain an injective module M such that every module 
in A has an injective resolution, all of whose terres can be embedded in M, and 
every module in C+ h an injective resolution ail of whose terres can be elnbedded 
in (M). By [1. Lemma X.3.2. p. 101], there exists a silnple cofree functor F from C 
to cochain complexes in  wih basis ((M), V) for some injective I', and natural 
transfornmtion  : H'(F) + R' which, on Ca, is a natural isomorphimn. The 
construction of T and the proof of the result now follows exactlv as for [1. Theorem 
X.5.3, p. 107].  

Theoreln 6.3. Let C be a Hopf algeb'ra over . a'td let A be a rtgkt C-comodule 
algebra with B = A c. Suppose A. B ad C are .-projective and tbat A/B is 
C-Galois. Let A be tbe category of left A«-modules. C the categorg of right C- 
modules a'd let 73 be tte category of .ff-modules. Let c : .,4 -- C otd l, : C -- 73 
be tbe flmctors defited bg d)(X) = X  atd /,(}-) = }-c .for X  A and }  C. 
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Suppose A/.¢I is .-pvojective and that A is botb left and right B-fiat. Suppose 
I 
F = (F, F-, I', 7') and F' = (F', F,, I-, 7-) are spectral .seque'nce co'n.structors 
for (¢, ). Tttea F and F  cottstruct caoztically isomorpbic spectral sequences fTom 
the E2-1evel o,ward. 

Pro@ The mgument of [1, Thcorem X.5.4, p. 109] al»plies unchmged. 
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FORMALITY IN AN EQUIVARIANT SETTING 

STEVEN LIIJLYWHITE 

ABSTRACT. ,Ve define and discuss G-formality for certain spaces endowed with 
an action by a compact Lie group. This concept is essentially formality of the 
Borel construct ion of the space in a category of com n|utat ive different lai graded 
algebras over / ---- H*(/3G). These results may be applied in computing the 
equivariant cohomo]ogy of their ]oop spaces. 

1. INTRODIICTION 
In this paper we consider G-Sl)aces and give formality results for them in an 
equivariant category. More si)ecifically, given a G-space 31. we discuss fornml- 
itv of the Borel construction EG x G 3[ or, eqlfivalently, formality of the com- 
plex A(11I) of equivariant differential ferres. However, in the equivariant settiug, 
the mai) 111 -- {pl.} is rei)laced 1)3" EG XG .I -- BG, and consequcntly ail the 
comnmtative differential gra(led algebras involved are naturally R-algehras, where 
R = H ° (BG). Thus formality may be considered in the category of comumtatiçe 
differential graded R-algehras. We shall also consi(ler the augmented case, corre- 
sponding te eqlfivariant base I)oints, which are the saine thing as fixed points of 
the group action. We should like te call a G-space 11I %quivariantly formal'" when 
its Borel construction is formal in the above sense. However, the terre "eqnivari- 
ant formalitv" has corne te be used te describe the degeneration of the spectral 
sequence of the fibration JII -- EG xE 11[ -- BG, owing te the i)ervasive influence 
of [11]. Thus xve shall adopt the terminology "G-formal" in this paper. 
We give seine general results concerning G-formaliy of products and wedges and 
reductions te subgroups. This is followed by several examples of G-formal spaces, 
including compact Ki4hler manifolds and formal elliptic spaces, among others. Of 
course, we rnnst make appropriale assumptions on the G-actions of these spaces for 
the results te hold. 
As an application of these results, we compute the equivariant cohomology of 
loop spaces. (If 11I is a G-space, then se is the loop space of 11I in thc obvious way.) 
Our motivation cornes frein considering the cohomology of symplectic quotients of 
loop spaces, sec [18], although the results are of general topological interest. 
shall use an "'equivariant" bar complex te compute the equivariant cohomology 
of the loop space. If the G-space 11I is G-formal, then the bar complex, which 
is generally a double complex, loses a differential and becomes a single complex. 
allowing for seine casier calculations. In the last section we compute an exmnple. 
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In an appendix, we discuss bar colnplexes and Eilenberg-Moore theory concern- 
ing the pull-back of a fibration. \Ve also consider equivariant versions of these 
results, which are used in several of the proofs in the lllain body of the paper. 
In what follows, we shall generally assulne that G is a compact. COlmected Lie 
group and that all spaces are colmected. Whenever we need to use the localization 
theoreln iii equivariant coholnology, we shall assulne that the spaces under con- 
sideration are of the holnotopy type of filfite-dimensiolml G-C\V-colnplexes, and 
flu'therlnore that they have finitely lllallv COlmective orbit types, lnealfing that the 
set {[G°] ].r e /I} is filfite, where Gx is the stabilizer subgroup at x. G°x is the 
comected COlnponent of the identity, and [G°] denotes the set of conjugacy classes 
iii G. This latter condition is autolnatically satisfied, bv the way, if ,I is colnpact 
or if G = S 1. 
I would like to extend lny appreciatiol to Chris Alldav. who took the tilne to read 
the lnalmSClipt and offel'ed advice on several key points. In particular. Proposition 
-1.7 is due to hiln. 

"2. /«C)Ç.A AND FORMALIT5 
In this seçtion we recall SOlne ilnt»ortant facts al»out the category of COlmnutative 
differential graded algeblas, the notion of forlnalitv, and the colmectioll with ratio- 
nal holnotopy theory. \V shall assume ibr now that out algebras are /«-algebras, 
where t" is a field of charaeteristic zero. \Ve shall denote l»y/«CT)Ç ° the category of 
COlmnutative diflerential graded /¢-algebras that are concentrated in non-negative 
degrees and have a differential that raises the degree by one. \Ve assume further 
that H°(A)  k, for ail A in hC7)91 '. \Ve shall denote bv /«C7)9 the category 
of algebra8 in /«CT)ÇI ° that are auglnelted over t" (i.e., there exists for each A a 
lnap  " ,4 --/«, with/« COlCentrat.ed in degree zero), together «ith auglnentation- 
preserving lnaps for morphislns. \Ve shall ca]l an object of hCT)ÇI (resp. kCT)Ç.A °) 
a/,'CDGA (resp. kCDGA°). 
We recall Quillen's abstract approach to homotopy theory, [22], [23]. He begins 
by defining the notion of a closed model category. A closed lnodel category is a 
category, C, with 3 distinguished classes of lnorphislns, called cofibrations, fibra- 
tions, and weak equivalences, which satisfv a nulnber of axiolns. The holnotopy 
category, Ho C, is defined to be the localization of C with respect, to the class 
of weak equivalences. Quillen introduces a notion of homotopy and shows that 
Ho C is equivalent to the more concrete category ho C which has for its objects the 
cofibrant/fibrant objects of C, and for its morphisms the homotopy classes of maps. 
We point out the importmlt fact. that two objects X and }" in Ho C are isomorphic 
if and onlv if there exists a chaill (in C) of weak equivalences 

(1) 

In [4] it is shown that the categories kC7?(ï,,4 ° and kCDgI are closed model 
categories where the weak equivalences are the quasi-isolnorphisms (maps that ill- 
duce an isolnorphism on cohomology), fibrations are the surjective morphislns, and 
cofibrations are lnaps that satisfy the following lifting condition: a lnap f is a 
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cofibration if for everv commutative diagram 

with p a fil»ration and weak equivalence, there is a mal» ffoto }" to 1: making the 
diagram commute. (Actually, in [4], the authors do hot assmne that H°(A) 
for all algebras A.  have includcd this assmnption for case of l>resentation, but 
the difference is slight.) 
Given a closed model category, C, with initial object ,, an object B is called 
cqfibrat if the map   B is a cofibration. B is called a cofibr«mt model for 4 
if B is cofibrmlt and there exists a weak equivalence B + A. It follows ffoto the 
axioms for a closed model category that everv ol»ject in a closed model category 
has a cofibrant model. Moreover, therc are various lifting and homotol>y results 
associated with cofibrant algebras: see [4], section 6. X> mention one here. If 
ç : Bx + B2 is a quasi-isomorl>hism, an<l we have a mat> f : A  Be with A 
cofibram, then there exists a lift f : A + Bt such that çf oe f, where m denotes 
homotol»y. 
Note that L'CÇM is I>ointed with point o]»je<'t L'. The homotopy groups of a 
kCDGA A are defined to be 
def (A/(4)  ) 
nA = H , 
where , = ker, for  : A  k a giveu auglnentation of A. 
If f : B  B is a weak equiva]ence of cofibrant kCDGA's, then f. : 'B 
'B is an isomorphism. Tlms, if we define R'(A) a (B) £r B a cofibrant model 
of A. then fl'(A) is well-defined up to isomorphism. Moreover. if f : Ai  A2 is a 
map of kÇDGA's, then f induces a unique homotopy class of maps f : Bt 
for fixed choices of cofibrmt ttlodels Bi. B2 of .41, -42. respectively. It follows that 
there is a unique mal» f. : H(A1)  H(A2). Thus H is flmctorial, and different 
choices of cofibrant models yield naturally isomorphic such functors. 
In kCÇM, there is a special class of cofibrant models called minimal models. 
A minimal model of an algebra A is defined to be a. cofibrant model, ,  A. 
that is connected (0  k), and such that the induced differential on /(«]2 is 
zero. It can be shown that each algebra in #CÇM has a minimal model, unique up 
to isomorphism. If BI is a path-connected topological si)ace, the (pseudo-dual) 
homotopy groups of AI are dened to be H'(M) d fl,(A.[AI) ) = (,), where 
 is a minimal model for A'(M). Here, A'(M) denotes the Sullivan-de Rham 
complex, which is a QCDGA; see. for example, [3] for the definition. If M is a 
smooth manifo]d, we mav also use the ordinary de Rham complex, taking k to be 
Halperin has explicitly identified the cofibrations (and hence cofibrant objects) 
in #CÇM. Çofibrations me the so-called KS-extensions, and the cofibrant objects 
are the KS-complexes. Since these llotiotls will be important to us, we give their 
Definition 2.1. A map f : A  B of kÇDGA's is said to be a IçS-e'lesion if 
there exists a well-ordered subset  C B  = {a'o}, Sllch that .4 C I (£) 
is an isomorphism of commutative graded algebras, where (£) dent»tes the 
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graded comnmtative algebra on the set E, and the map is induced bv f and the 
inclusion of E C/3. Identifying/3 with A  (E). the differential on B satisfies 
(1) dB(a @ 1) = dA(a) @ 1, 
(2) dB(1 
where E< = {x [[ < }. If E also satisfies deg(x) > 0 Vx Ç E, and deg(x) < 
deg(x)  fl < a, then f is called a minimal A'S-extension. If A = k, then we 
replace the word "extension" by the word "complex" in the definition, obtaining 
the notion of KS-conple. (A minimal KS-complex is the smne thing as a minimal 
algebra, defined above.) A minimal KS-extension in which A is also minimal is 
called a A-'minimal A-e¢tension. Note that in a (minimal) KS-extension. A(E) is 
a (minimal) KS-complex, with differential such that ¢  1 :A  A(E) + A(E) is a 
map of kCDGA's, where ¢ is the augmentation of A. Moreover, all of these maps 
mav be ruade compatible with augmentations. 
If A is a kCDGA, then its cohomology, H(A). may be considered to be a kCDGA 
with zero differential. 

Definition 2.2. ,| is said to bc for'mal if A  H(A) in Ho(kCl)Ç.A). 
It is easy to see that this (lcfinition is eqnivalent to the following two. 
Lemlna 2.3. Consider tbe cate.qory kCÇ. The following are equivalent: 
(1) A is formal. 
(2) Theve is a diagram 
A  B  H(A), 
 where the maçs are weak equivalences and B s a coflbrant modelfor A. (In 
particular, we may pick B to be minimal.) 
(3) There is a chain of quasi-isomorphisms 

A *- A1 -- A2 *- "" - A, -- H(A). 

This theory has an important application to rational homotopy theorv. It turns 
out that the homotopy category of rational finite Q-type nilpotent spaces is equiv- 
alent to the homotopy category of the fllll subeategory of QC/PÇM consisting of 
algebras A with HA of finite type, [4]. Thus we may "do'" rational homotopy 
theory in a category of differential graded algebras. As an example, if X is a 
path-commcted, simply-connected topological space of finite Q-type, then there is 
a natural isomorphism 
II'(A°(X))  Ho,n(r.(X) ®Q, Q), 
where A°(X) is the QCDGA of Sullivan-de Rhmn differential fonns on X. If X is a 
smooth manifold, the saine statement for homotopy groups holds if we use instead 
the de Rham algebra A°(X) and replace Q coefiïcients bv NI, or C. There is hot a 
corresponding equivalence of homotopy categories over NI or C, however. 
A path-connected topological space is said to be formal if its Sullivan-de Rham 
algebra A°(X) is fonnal. If X is a smooth manifold, we may use the de Rham 
algebra and real or complex coefficients. However. a well-known result in rational 
homotop.v theory states that formality over NI or Ç implies fonnality over Q: see, 
for exmnple, [15]. 
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Formal spaces include comI)aCt Kfihler mmfifolds and many homogeneous spaces, 
including compact globally synlmetric sl)aces. Produçts, wedges, and Comlected 
smns of formal spaces are agaill forlnai. The top«,logical consequences of formality 
include the vauishing of ail Massey prodnçts. Moreover. the rational homotol)y 
type of suçh a space is deternfined solely 1¢« its cohomology algehra (at least for a 
large class of such spaces). 

3. RC)ç AND G-FORMAIATY 

In this paper, we shall Ie concerned with equivariant versions of standard for- 
mality resuhs. Let G he a compact, COmlected Lie group. Then H'(BC:h) is 
isolnorphic te the hCDGA freelv generated bv a finite number of generators of even 
degree.  shali denote H de H'(BG).  dt, fine the category CÇM ° te be 
the category of comnmtative differential graded -algelras.  shall contimw te 
assume that H°(A)  k for ail algebras A. Tlms, we ohtain a faithflfi forgetfld 
fimctor ff'oto RCÇA  te kCDÇA".  also define RCÇA te l»e the category of 
commutative differential graded R-algelras augmeuted over R. Conlposiug aug- 
mentations with th¢' augmentatiol R  h. we get a faithfifl forgetfid fimctor fiOnl 
RCDÇA te L.CDÇA. 
It is a standard result that if C is a closed model category and B is au object 
of C, then the "'over category" C/B whose o]»jects are nlaps X  B and whose 
lnorl)hisms are commutative squares of the type 

X f  " 
1 1 

may be given the structure of a closed model category with the following definitions. 
Such a morphism in C/B will be called a fil»ration, cofihration, or weak equivalence, 
if the map f : X  I" is such in C. A similar statement holds for the %roder 
category", BC. See [7] for these and other results about closed model categories. 
Thus we see that both RCDç  = RkCDçA  and HCDç = RCDç/H are 
closed model categories. Moreover, the sinlplicial category structure on kCDÇ  
defined in [4], section 5. induces a simplicial category structure on RCDA" and 
HCDÇA in such a way that the results of [4], section 5, suitably modified, hold for 
these categories as well (cf. [22], II.2. proposition 6). Frein this, it follows that 
the homotopy results of [4], section 6, suitably modified, hold for RCDçA ' and 
RCDç as well. 
Definition 3.1.  shall sv that an RCDGA (resp. RCDGA ) A is fowml if 
A  H(A) in Ho(RCDç) (resp. Ho(RCDç)). 
If a filnctor j : C  C t)etxvn two closed model categolies preserves weak 
equivalences, then A"  l" in Ho C iluplies j(X)  j(l') in Ho C. Thus if an 
algebl'a A is formal  an RCDGA, then it is formal as an RCDGA , and  a 
kCDGA, etc. 
Sul)pose a smooth lnanifold BI has a smooth action of a compact Lie group 
G. The equivariant coholnolo&v of I mav be computed bv means of the Cartan 
complex of equivariant differential ferres: A;(M) = ((S*  A'(M)) G, dG) where 
the different.ial, da, is zero on S*, and for o.  A" (M), dGt = de-uitx, o, where 
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the {Xi } are fundamental vector fields of the action correspmding to a basis of l- 
and the { u, } are the corresponding algebra generators of S*. which are given degree 
two. If 31 is just a topological space, we llltly compute the equivariant cohomology 
of 3I bv means of the QCDGA A(3I) of [2], when G = S t. Alternatively, ce 
could use thc de Rham algebra of the Borel construction. A'(EG xa M) when M 
is a lnanifold, o1" the Sullivan-de Rham algebra of the Borel construction when M 
is hot a manifold.  shall let A;(M) possibly denote any of the above kCDGA's. 
leaving it to the reader to interpret which mdel one prefers to use,  xvell as whid 
gromM fiekl k. FLra çOlnl)rehensive tleatment of equival'ialt de Rham thcory, see 
[:]. 
Using either model, it is olvious how to oltain an R-algelra structure on A(M). 
i 
If is indu«cd 1,v H  ,t(pt.)  A(M), »vhere the fil'st map is a choice of 
mildlnal model for A(pt.) in kCÇ, and the second map is induced from the 
map 1I  {pl.}. If vc use the Cmtan models, then the algebras A(M) are 
manifestlv augmented (ver H whcn the group action bas a fixed point. This is 
lecause in the Cmtan model, A;(pt.) = , and thc inclusion of a fixed point gives 
a mal» A;(II)  A;(pt.) = . [h,wever. if we use the Sullivan-de ham complex 
ff the Bnel «mstruction. then A(pt.) = A'(BG) ¢ R. Thus we nmst use a quasi- 
isomorphic «omplex that is Slnaller and augmented over . In [1], Allday shows that 
the «Oml,lex ,/-()is quasi-isomorphi« to A;(M), where ,/ A(3I)  A(pt.)is 
induced by the in«lusion of a fixed point into [, and  is embedded in A(pt.) via 
i as above. Clearly. tl-() is augmented over R. and is flmçtorial for eqtfivariant 
maps of G-spaçes. We shall almse notation and continue to write A;(M). even 
when we may really mean I-(R). 
Let ÇTO denote the category of l)ath-comected topological G-spaces with 
morphisms the equivariant maps. Then the under category {pt.}ÇçO consists 
of "%ased G-sl)aces", which is the saine thing as G-spaces with non-empty fixed- 
point set and a choice of base point in the fixed-point set. Then A(-) gives a 
flmctor from ÇçOç to RCDÇ  and from {pt.}ÇçOç to RCDç. 

Definition 3.2. We shall sav that a G-space .I is G-formal if A(3I) is formal 
as an RCDGA". A G-space M with equivariant base point p (i.e., a choice of fixed 
point p  3I a) is G-formal af p if A(3I) is fornlal as an BCDGA. where 4(M) 
is augmented via the inclusion of p into 

If we continue to define a minilnM model of an BCDGA as a connected cofibrant 
model M for whMl the induced diflbrential on ker e/(ker ) is zero, where  is an 
augmentation over . thell there moEv hot be a minimal model for everv algebra 
in C. An exmnple is .ç actiug bv rotations of .q2 about an axis. It is easy 
to see that there can l»e no minimal model for A. (S ) in BCDÇA. However. 
there is a fairly canonical choice of cofibrant model for an CDGA. Let   A 
be an RCDGA . Then the map R  A, viewed in kCÇM, mav be factored 
as     ,M  A with the first map the inclusion, the latter lnap a quasi- 
isomorphism, and M a minimal KS-complex, [14]. Note that the differential on 
R @ M is hot the tensor product differential: see the definition of a KS-complex 
(Definition 2.1). The map R  R G' . is a nfinimal KS-extension. in particular a 
cofibration in kCçM, and hence we sec that R , . is a cofibrant model for A in 
RCÇM . Suppose A is, moreover, an algebra in RCDÇM. and let e : A  R be 
its augmentation. Then composing B ï .  A G B gives an -augmentation for 
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R  .A4. Thus, R . .,M/ becolnes a cofibrallt model fbr 4 in the categorv RCTbÇ.A. 
As defined, itis unique up te isonorphism. 
Fol" those algebras of the tbrm A(./II) arising ri'oto a group action on the spa«e 
31. this cofibrant model is more explicitly given b) the Grivel-Halperin-Thomas 
theoreln, which states that there is a c(mmmtative diagram 

(z) 

i 
R --* R '.in,..t --, 

asso«iated te the filtration 11I -- EG xc M -- Bd7. where .A4 is a minimal model 
br M. and the ],otom row is a A-minimal A-extension. see [121, [14]. 
Definition 3.3. && shall rebr te R .   the G-mod«l of A, or just simply as 
the G-mod«l of M. when A : A(M). 
Sometimes we mav ehoose te dent,te it lx _( df B. . Note that R , 
may tdl te lin minimal as a k('D(',A. 
Folloving [1], [3], given a lmth-colmectcd G-space 1I with equivarimt I,ase point 
(i.e., a fixed point) p. the equivariant (pseudo-dual) k-lumotopy groups are defilmd 
te 
(3) Hâ,p(M) a '(R ._ Al)= H'(kere/(kel'e)2), 
where ge : R  . + R is the R-algebra augmentati«m induced ])x the inchisiol of 
p into M. as above. The assigmnent (31. p)  (R..ç.«) gives a flmctor ff'oto 
{pl. }çOç te Ho(RCDçA). and t he equivariant pseudo-dual k-homotoW group 
are fimctorial as well. Note that if M is G-forlnal. then the equivaliant pseudo-dual 
k-homotopy groups are determilmd by the equivariant coholnology ring of 1I. 
The following lelmna is uscflil for comparilg the equivariant pseudo-dual k- 
holnotopy groups te the oldinarv pseu&-dual k-homotopy groups of the Borel 
construction. 

Lemlna 3.4. Let A he an RCDGA «td let R. .A4 t,«- the G-model for .4. Then 
R ,. )., is minimal in kCDgA. 

Pro@ We have the augmentatiol g : R ,.-_ a'vl --, R. vhich is a lnap of RCDGA°'s. 
The differential. D. on /7 ._a .A4 satisfies D(r. ' 1) = 0. for r  R, and generally has 
the brln D(1 " oz) = r C 1 4-  ri  ai 4- 1 ._ç da, where a, ci,  .,. r. ri  R with 
deg(a).deg(a).deg(r).deg(r) > 0, and where d is the differential in «VI. New. 
I) = eD(1 ® a) = "r + rie(ai) + e(da). Since da  (+)2. and e is an algebra 
map, it bllows that.  ris(ai) + e(da)  (17+) 2. Hence, we nmst bave that r = 0, 
and it fbllows that 17 .-> .k4 is mininml. [] 

As ait exalnple, the pseudo-dual k-holnotopy groups of the Bolel constrnction of 
S t acting 011 '2 do hot distinguish the trivial action froln a standard non-trivial 
one, whereas the equi'a.riant pseudo-dual /-honlotopy groups do. 

4. C-ENERALITIES CONCERNING (ïLFORMALITY 
In this section we give some basic results about G-formalitv. ilwluding reduction 
to subgroups and the G-brlnalitv of products and wedges. 
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\\ I)egin 1)y noting that formality in the category HCT)ÇA ° is equivalent to for- 
malitv in kCDGA, hl genera.1, for two R-algebras A and B, A  B in Ho(kCÇA) 
does hot imt)ly that A  B in Ho(RCÇA°). Nevertheless, we have the following. 

Lemma 4.1..4s.s'ume that R -- 4 is an RCDGW' and that we give H(A) the 
j* 
R-algcbra structure R -- H(A). Then A is formal in I,'CT?gA if and onlg if A is 
formal in RCDÇ.A ('. 

Proof. If 4 is tbrmal in RCT_GA", then it xvill be so in 1,'CTÇA, as we have noted 
al»ove. Let us now assume that A is formal in/,'CY)ÇA. Let ,'( lin a minimal model 
for A and let R 9 .'v/be the G-modvl for A. Thon we have a «onmmtative diagram 
of/'( DGA s 

(4) 

.4 

Sin«e R, A is «ofibrant in kCI?ÇA, tllcre exists a nlap. which is ne«essarily a quasi- 
isomorphism, H.-o A -- N" making thc uppcr right square homotopy commute. This 
gives us a quasi-isomorphisn   R O Ad -- .&oe -- H(A). Thon the map 

(5) 

(*)- r)* 
R®,H(A) -« H(R®.Ad)---,H(A) 

is a quasi-isolnorphisln and a mai) of R-algebras. 

R«mark 4.2. \Ve note that this is hot true for lnaps, however. That is, if f  A -- B 
is a nmp of R(DGA s, and fis formal as a mal) of/,'( DGA s. then f need hot be 
a formal map of R(DGA s. 

In the category RcT)gA, fi)rlnality is a. conccl)t distinct fronl formality iii the 
category kCg)ÇA, hl fa«t, it is easy to see that .,I is G-fornml a.t p if and only if 
the map i " BG ---, EG x M is a formal map. where i is the map indu«ed bv the 
in«lusion of p into 3I. 

Definition 4.3. Sui)pose that G acts on a space M. Then the Serre spectral 
sequence associated with the fibration 31 - EG xa M -- BG is the saine as the 
spectral sequence (from E2 onwards) obtained ri'oto the G-lnodel R ® ,/ via the 
filtration ffP = R >-p ® flA. If this spectral sequence degenerates at the E2 terre. 
then [11] refers to 5I as being equivariantly formal. For obvious reasons, we wish 
to avoid this terminology; however, to conform as well to current trends, we shall 
sa3 that 31 is er when this spectral sequence degenerates at the E2 terre. 

Proposition 4.4. Let G act on a space M. Suppose that K C G is a cloaed. 
connected subgroup. If 5I is G-formal at p (or G-formal) and er. then 31 is K- 
formal at p (resp. K-formal). 
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Ptvof. We fil'st consider the case where 'I is G-folnml at p. The inclusion h" C G 

induces a pull-lmck diagraln 
Eh xtç 111 -- EG xG 111 
i 
B Iç -- BG 

We shall denote H'(BG) by Re;, a.nd sinfilarlv for Hl. If we are using the 
Cartan complex of equivariant differential forms, then there is no lnobleln with the 
proof. If ce are using Allday's construction, q-l(H), as notated al»ove, then we 
face file possibility that this construction may hot be flmctorial with respect to 
changing the group. This is because there may hot exist choices of minimal models 
so that RG -- Rh -- .t'(BK) conmmtes with Re, -- .I'(BG) - -I'(BIç). Then 
there «ould hot exist an iuduced map q-  (R) -- q- l (/ï't,)- 
This problenl may be circumvented by the following procednre, as pointed out 
to us by C. Allday. ('onsider file diagranl (6). Let f" &,note tlle mal»ping cylindcr 
of the top row, and Y the malpillg cylinder of the bottom row. Then we have a 

in which the nmps jl,fl induce SUljections on differential forms and the maps j2,. 
induce quasi-isonlorI»hisms on differential forms. Il is easv to show that we nmv use 
17- to form the complex q-l(/ï'G), as discussed in section 3. and that this COlnplex 
will be quasi-isomorl»hic to ,4(1I), and G-formal at p if 31 is G-formal at p. 
Moreover, we now may obtain a COlnlmltative diagram 

A'(Y)  A'(BIç) 
in which the vertical arrows are quasi-isomorphisms, since the inap A°(Y) -- 
A°(Btç) is onto. This follows bv the result for RCDg,A. which is the analog of 
By Lemma A.1 of the appendix, there is a qui-isomorphism of kCDGA's 
(9) (A'(Blç),A'(BG),Ab(M))  A-(M), 

where we are abusing notation in the event that we are using Allday's construction. 
Then, in either case, we obtain a quasi-isomorphisln of R/-CDGA's 

(0) 

[(RI«,Ra, A'a(M))  A-(M). 

The bar complex (10) is an R/«-algebra via the R/« factor, and has an RK-augmen- 
ration given by e(rK,ct) = rKï(eG(Ct)), where r/ ¢ Rê,', et ¢ A.(/I), and ea " 
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.4;(I) -- Rc is the augmentation of I for the action of G. Bv the asslnp- 
ti«m of G'-formality, we get a conmmting diagram whose vertical arrows are quasi- 

isomorphisms: 

(11) 

Rb  RG -- 
Riç  RG --" H(ll) 

Then xxe obtail, lhe folh)wing sequence «)f lllaps, which are st'en to be R,CDGA 
«luasi-isolmn'l»hislnS i W standard cmnlmrison theorems tbr their associated Eilen- 
ierg-g Ioore spec tral seq,ences: 
(12) B(R,ç. Ra, .4;(.I))  B(R,,. Ru..a(3I))  (R/,. Ha. H;(M)). 
Now the lmr «omplex B(Rt,. RG. ff;(.ll)) has mly lle singlc difl>rential . alld 
comt,utes TorRo (Rb. H(3I)). Since 1I is cf. Hç.(l) is a free RG-module. Hence 
we have thal (Rh,HG,II(3l)). is acvclic in lmr degrees greater than zero. and 
t ho projection to c()h()lllology 
(13) (Hr,, Re, H5(3I)). + ( Rç, Re, 1]:;(31))o 
 H,-(M) 
is an RçCDGA quasi-iSOlnorphism. 
The case where we consider 31 to be G-formal in the category HGCÇ ° is 
similar. 

Corollary 4.5. Let G act on a space 31. Suppose that M is G-formal at p (or 
G-.formal) rmd er. Then 1I is formal in 
Pro@ Just take K to be the i(lentitv sui)gr()ul) iii Proposition 4.4. 
emark 4.6. If we use Remark A.5 of thc apI)elMix, then we can see that a lx-nlodel 
fbr M is given by Ra(Rh. RG,G(l)) : Rb R (I). 
In line with the general thelne of considering maximal tori in compact. COlmected 
Lie groups, we have the following face. which is due to Ç. Alldav. 
space M is of fiite tpe if H'(M) is a finite-dilnensional k-vector space for ail i. 
Proposition 4.7. Let G act on :I. a«td let T C G be a ma.rimal torus. If M is 
G-formal (G-formal at p). then M is T-formal (resp. T-formal af p). Moreover. if 
M is a space offlnite tpe. p  M . and I is T-formal at p. then M is G-formal 
at p. 
Pro@ X can aheadv see tlmt G-forlnal ilnplies T-formal by the proof of Propo- 
sition dA. XX> only need the fact that now HT is a fi'ee Ho-module. which follows 
ri'oto the well-known fact that as Ho-modules. RT  Ho Ç H'(G/T). 
Showing t.hat T-formel al p implies G-formal at p mv be achieved by imitating 
the proof that A  K being forlnal ilnplies A is forlnal, for K an extension field of 
k. whi«h is corollal'V 6.9 of [15].  Olnit the details, bnt mention the setup. First. 
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we see by I{emark 4.6 that a T-model for A(AI) is given bv/r T (-RG .A4c(M) with 
differential 1 C, De, where De is the difl'ereltial fiJr the G-mode| .Me(iii). Thus it 
suffices to show that if/-ff ®ha ,(/11) is formal as au RTCDGA. then .M(M) 
is forlllal as ail RG('DGA. 
It turns out t|lat the coustl'uctions of bigra(led and filtere(l models of the rçlevaut 
algebras, as in [1.5], givc lUodels iu the category HCTÇ4. Thç proof of corollary 
6.9 may bc imitated xvitllout too llltlch difficulty. [] 

Proposition 4.8. Supl,oSe that X and  arc G-spa«es. both of whi«h are G-formal 
(or as.smne X is G-fo'rnml at p and " is G-formol at q). and suplo.e that one 

or both of them is 
diagonal a«tiou of G. 

Proof. The pull-back diagram 

(14) 

G-fin'mal (resp. G-formal at (p. q)) for the 

x --, b,t.} 

gives rise to a lmll-lmck diagram 

EGxG(XxY)  EGXGI 
EG xG X  BG 

(5) 

  O 
Then we o])tail a.li /( DGA qnasi-isomorphism 

by Lemlna A.3 of the apt)endix. If X and ] both have fixed points, then so will 
their product X x }'. In that case, 0 is a quasi-isolnorphisln of RCDGA's bv Lemlna 
A.3 of thc appendix. FurthCllnOlC, 
(17) - " " - " " 
B(A.(X)...a.())  Ai, 
" B(AG(X ), ,({pt.}), ) 
is an RCDGA ° (RCDGA) quasi-isomori)]fiSlU. Since X aud  are G-forum.l. we 
get RCDGA ° (RCDGA) quasi-isomorphisms of bar cmplexes 
(H&(X),R.H&()) 
by standard arguments comparing the associated Eilenberg-Moore spectral sequen- 
CeS. 
Siuce one or both of X. ) is cf. just as in the proof of Proposition 4.4. the bar 
respect to the bar grading, and the ln'ojection to its cohomology is an RCDGA  
(RCDGA) quasi-isomorphism. 
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Proposition 4.9. Let X md I" b« G-spaces u,hose fi:red-point sers are non-empty. 
Picking base points p E X G and q  l'G. we may form the wedge X V I" along these 
base points. Theu G acts on A V }'. If X is G-formal ai p and 1" is G-formal at 
q. then X V }" is G-formal al the join of p and q. 
Pro@ Let ex,e denote the augmentations of equivariant differential forms, and 
let ix, if denotc the inclusions of X, 1" into X V Y. Then Mayer-Vietoris gives a 
short exact sequence 
Thus ix + i- induces an isomorplfisn 
R 
Moreover. since ex, say, induces a surjection in cohomology, the associated long 
exact sequence splits into short exact sequences, and thus 
(2O) nS(X V )  n5EE(X ) e bO')" 
Since ç and k are G-formal. we have maps A(X)  a(X)  H(X) which are 
quasi-isomorphisms of RCDGA's, and similarly for I'. So we have a commutative 

diagram whose rows are short exact sequences: 
(21) 
o  Ab(x) ¢ A.(',) --- Ab(X) 

,4c(Y)   --, o 

Then we obtain maps between the associated long exact sequences in cohomology. 
By the 5-1emma, it follows that the maps 
(22) A,(X) eR A,(Y) 
are quasi-isomorphisms. It, is easy to check that these maps are compatible with 
augmentations and the R-algebra structure, so are RCDGA quasi-isomorphisms. 

5. EXAMPLES OF G-FORMAL SPACES 
In this section we give some examples of G-formal spaces. 
5.1. Compact Kihler lnanifolds. Let 3I be a compact Khler mmfifold, and 
G a compact, connected Lie group acting on 3I by holomorphic transformations. 
 introduce equivariant holomorphic cohomology groups. Since M is a complex 
manifold, the complex-valued differential forms on M are bigraded in the usual 
way. ë shall denote Sg* @ C by simply Sg*. Then we define the eqtfivariant 
Dolbeault. cohomology to be the cohomology of the complex 
(23) ([$9" Ce AP"(I)] ° ;  +  ,z,). 
Here Zi is the holomorphic vector field on M which cornes about by splitting the 
fundamental vector field Xi = Zi + Zi into its holomorphic and anti-holomorphic 
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colnponents. The gellerators ui E Sg* are given bidegree (1.1). The operatol's act 
in a Silllilar way as for file ordinarv equivariallt cohonlology. \\k' shall denote the 
qth cohollology of this complex by H'q(AI). 
The following theoreln was proved in [17] and ildependently established in 
Theorem .1. Suppo.s'e that Al is a compact KShler maifold edowed with a 
holowophic action of a compact, covected Lie 9roup G. and sui»pose that I s ef 
for the action of G. Then M is G-Jbrmal. IJ" M a 
fized point. 
Pro@ The Cal'tan COlnplex is (A(M),da)= ([S* N.4"(3I:C)] 
Let Xi = Zi + Zi be the splitting of the flmdalneltal vector field Xi into its 
hololnoll»hic and anti-hololnorphic parts. The differential d = O + 0 also splits. 
Hence we lnay split the equivariant differential as d +  utx, = (0 +  tz,) + 
(0 +  ut,). The COlnplex [Sg*@ A(M;C)] a is bigradcd by giving 
bidegree (1.1). and taking the usual l»igl'ading on A* (1I; Ç). It is easy to show that 
(24) ([Sg*A'"(M:C)]a ; (+u,z,),(O+u)) 
is a first qua.drant double çOlnplex. Accoldingly we have two canolfical filtl'ations of 
this COlnl)lex.  claire that the spectral sequences COl-lespondilg to both of theln 
degenel'ate at the E1 terln, and lllOl'çover are n-opposite, memfing that 'F p 
H' for p + q- 1 = tt. Formalitv for 4(M) then follows owing to the results in [6], 
sections 5 and 6. 
Let us consider the filtration iii which we take  +  uitz, coholnology first. 
This is the Dolbeault equivarimt coholnology defined above. It itself fOl'III 3 fil'St 
quadrant double complex with the two differentials  and  uitz,. Let us filter 
so that we take the 0 coholnology filSt. Then the E terin for the equivariant 
Dolbeault COlnplex is (additively) 
(25) H([S*  4"(M)]a:O)  [H(Sg*  A'(M):O)] a  (Sg   H(M)) a 
(26)  (SO*) a  H3(M)  H'(BG)  HO(M ). 
Now bv ordinary Hodge thcory fbr COlnpact Khler manifolds, this lt is isolnorphic 
to H'(BG) H'(I). But now there can be no further non-trivial differentials 
iii the spectral sequence, by the aSSUlnption that M is ef. This result follows 
mmlogously for the other filtration, which is just the COlnplex conjugate of this one. 
Furtherlnore, it is easy to see that the two filtrations m'e n-opposite. 
Hence we have a "OGOG-lelnlna'" for the equivariant differential forlns, where we 
mean by 0G the equivariant Dolbeault opcrator as defined above. Forlnality follows 
via the sequence of CCDGA quasi-isomorphisms 
(27) Ab(M)  ker(Oa)  Ha(M ), 
which are the inclusion and projection, respectively. These maps are lnaps of R- 
algebras, and moreover, it follows that for equivariant holomorphic maps between 
M and N, we get a commutative diagraln linking the sequence (27) for .I to the 
analogous sequence for N. In particular, if the action of G on M bas fixed points. 
then the inclusion of one (chosen as an equivariant base point) gives augmentations 
so that the sequence (27) comlnutes with auglnentations. That is, 3I is G-formal 
in RCÇA. 
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Corollary 5.2. Sttppose that M is a compact K5hler manifold edowed witb a 
holomorphic action qf a compact, cone«ted Lie 9roup G. Assume that AI G # 
7Jen M is Gçformal at ang fi.red point. 
Pro@ Lel p G 3I . Let T c G 1)c a maximal torus. Then 3I  ¢ , and a theorem 
of Blanchal'(t savs that ;I is ef for the action of T: sec [9], Cai)ter XII. theoreln 
6.2. By Theorem 5.1. M is T-fi»rlnal at p. By Pl-opositiol 4.7. 31 is G-formal at 
p. 
Rcmark 5.3. The proof of Th«or,ln 5.1 implies an cquival'iant Hodge decomposition 
5.2. EIHptc spaces.  re«all tiret an elliptic space 1I   stmce su«h tlmt both 
tl'(M: k) and I" are finte-(limensonM k-vector spaces, whcre (M) = (I') is 
mhfima] mdel fi»r 1I. b shall nse the followh remflt of [19]. 
PropoMt}on 5.4 (Lui»ton). Let F  E   e  fibr«dion i which F s formd 
«nd «lliplic. and  is .fovmol che simpl.q-conne«ted. If th« Serre spech'al 
of the fihralion degenerot«s «t the E2 l«rm. the E is formd eIso. 
Theorem 5.5. Let 3I e az dlipti« G-poce If M is formel and er. the M is 
(;çformol. If ll G  . lhe zI is G-formol ci ez.q .red poinl. 
Pro@ We lmx-e the filtration ,I  SG XG :11  BG. Thon Proposition SA 
hnpfies t, lmt A.(,ll) i fin'mal as a k('DGA. çhe proof of Lupton's proposition works 
(a(iapthg to oto" situation) Lv finding a model for A(AI) of the form R.,ç that is 
bigraded a a kCDGA. Hcre, ç is the bgraded (nfinhnal) model of 3I. Elements 
of R arc in degrec zero for the second grading, so tha (R  -)0 =  - (,)0- It 
is shovn that wth respect to tlw second grading we have H+(H  ,) = 0. and 
hence the projection to cohomology 
(29) R., M  (R.)o  H(3I) 
is a quasi-isomorphism. Çlem'lv. this is a map of R-algebras. Ioreover. if 3I   
then this mat» commutes with the augmentations over R. This follows because first 
the mal» R ;-)  (R :-))0 commutes with augmentations. Second. since the 
augmentation e : R -   R is a map of RÇDGA's, e(do) = ) for ail . so that 
the map (R , ,)0  H(AI) commutes with augmentations. 
Corollary 5.6. Let 31 be at elliptie space. Suppose that a tortt T acts on 3I u,ith 
3I T  . Sui»pose fltrther that one of the «ompo#e#ls of the ç.red-pomt set, 
Pro@ Since AI is elliptic, it. follows (via localization and loca]ization for equivari- 
ant rational homotopy [3]) that each component of the fixed-point set is elliptic and 
k(AI) = (AI), where k is the homotopy Euler characteristic. But Halperin 
h shown that for elliptic spaces the conditions H °dd = 0 and k = 0 are equiva- 
lent. and moreover such spaces are forma/. Thus 0 = (3I) = (I). Hence 31 
is formal and H°dd(Al) = O. But this latter condition implies that 31 is ef. So we 
mav apply Theorem 5.5. 
Remark 5.7. Suppose G aets on a simply-connected space AI with non-empty fixed- 
point set. Then bv picking a base point in the fixed-point set, we obtain an action 
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of G Oll the Sl)ace of |,ased loops in 1I, deuoted tM. Since the cohomology of Lll 
is fret, we see that t]l will be G-formal if t/l is er. (Lupton's proof could be 
extended to this case, as well.) If G = T is a torus, and M is elliptic, then the 
condition that t]ll is er is equivalent to the G-model R ).A4(M) being mininlal in 
the category RCDçA: see [3]. 3.3.15. 

5.3. Miscellanea. Next we shall give a few extra examples of G-fornlality. 

Theoreln 5.8, Let 11 be a space u,itb minimal modcl « = /(I'). Sui»pose that 
d.r = 0 .for all x G I "'v«' such th(d deg(.r) < diln lI. Sui)pose fltrther that the civle 
S  = T acts on M. that M i.s @ and lhal cach comportent o.f the fixed-poinl set is 
formal ad satisfics H°ad(jl T) = O. Then ll is T-formal al an9 fixed point. 

Pvof. Since M is ef. the Serre spectral sequence for the fibration AI -- ETXT3I -- 
BT degenerates at the E2 terre. (Note that by the localization the«weln, this implies 
that M w ¢ .) Bv a standard change of basis argmuent, we lllay assllllle that in 
the T-mo(lel (R.,'t//, D) wc havc D.r = (). for x  I 'w such that deg(.r) < (lira 31. 
Let i : ll r '-- M &,note the inclusiou of the fi'<cd-l>fint set. Th('n wc have maps 
of/ï'('DGA's (actually, the algebras on the righ-hand side of the diagram d<) no 
satisy H ° = k, lmt t]tis will m)t l)resent any l)robh'lnS) 

(30) 

i 
A.(I) AI(M r) 
R ® cM(M)  R® 3A(3I r) 
/4(M) HT-(M r) t e t/'(M r) 

whel'e/t is a quasi-isonlorphisnl sinçe 3I T is folmal. Since 3I is ef. the nlap i* is 
an injection. We claire that hi(R ®.A4(M)) ç i*(H-(M)). Since the maps are 
algebra maps. it suffices to check this on alget)l'a genelatol's. Since M is er. the 
localization theorenl shows that i* is an isolnolphisln in degrees >_ dira M. Also if 
c  R-,M(3I) bas odd degree, then bi(o) = [k since H-aa(M r) = 0 bv assmnption. 
So it sutfices to check the claire on algebra generators of R N,/(,àl) of even degree 
less than dira M. Let o be such a generator. If o G R, theu the claire is obviouslv 
true. If o' G .A4(]l), then by assunlption Dc = 0. Then bi(o) = [i(cQ] = j([o]). 
Thus we have a map 

(31) 

which is a quasi-isomorphisn of RCDGA's. 

Corollary 5.9. Let ]I be a smtplg-connected space witk minimol model ](I/). 
Suppose that dx = 0 for all x Ç V « such that deg(x) < dira M. Suppose furtker 
that a torus T acts on M. that M is @ ad that each composent of the fi.red-poivt 
set is formal 
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Pro@ First of all, there is a subcircle S t C T such that 11I si 
of this circle ,S 'l  T induçes a imll-ba«k diagram: 

= AI r. The inclusion 

ES  X s 
B,ç I -- BT 
Since the action of T is er, the Serre spectral sequence for the fibration on the 
right degenerates at the E2 terre. Bul then the saine is true for the pull-back 
fibration. Hence lhe S  action is er  well. Now the result follows from Theorem 
5.8 and Corollary 4.5. 
Corollary 5.10. Let 3I 4 be a .sp,,ce such that Haa(kI) = 0 atd dimkl = 4. 
Eq»pose that a circle S  = T acts on M. Ten M is T-Jbrmal al ang fi«ed point. 
Pro@  have that H°aa(I) = 0, so that 31 is er. Then ai T  . Bv localization. 
H°aa(5l r) = 0. But path-comccted spaces xvith H  = 0 of dimension less than or 
(,qual to 4 are tbrmal; so each component of 3I y is formal. The result follows bv 
Thcorcm 5.8. 
Remark 5.11. A simple example of an Sl-space satisfying the conditions of Theorcm 
5.8, but which is hot Kfihler or elliptic, is the following. Let S  act on S 4 so that 
the fixed-point set consists of two isolated points. Exlend this to a diagonal action 
of S t on S 4 x S 4. Then, removing a ncighborhood of a fixed point, we may forln 
the coimected sure S 4 x $4S 4 X S 4. This manifold then inherits an S  action with 
6 isolated fixed points. It is hot elliptic, and not even synlplectic, since H  = 0. 
It is easy to check that it satisfies the conditions of Theorem 5.8. so is sl-formal. 
(This can also be seen by proving that the commcted sure (ruade in an equivariant 
setting) of G-formal spaces is again G-formal. which we have omitted.) 
 conclude this section with two examples that do hot involve the condition of 
3I being er. 
Lemma 5.12. Let M be a simply-co,nected compact manifold. Suppose that G 
acts freely on M ad dira G k dira I - 6. Then M zs G-formal. 
Pro@ Since G acts freely, M/G is a simply-comected manifold of dimension 6 or 
less. Hence M/G is formal [21]. So EG 
Remark 5.13. Suppose, iii the situation of Lemma 5.12, we have that dira M - 6 
rank(G). Let T C G be a llaXillml torus. Then by Proposition 4.7. MIT is a 
simply-commcted manifold of dimension greater than 6 which is formal. 
Lemma 5.14. Let M be a simply-connected elliptic space. Suppose that G acts al- 
most Jkeely on M (meaning ail isotropy groups are finite), and rank(G) = 
-X(M). Then M is G-fomal. 
Pro@ Since M and BG have finite-dimensional pseudo-dual rational homotopy, so 
does EG x a M, as mav be seen by considering the fibration M  EG x a 3I  BG. 
Since G acts almost freely, H" (EG x aM) is finite-dilnensiolml as well. Furtherlnore, 
(33) (EG xa M) = X(M) + x(BG) = -rank(G) + ra.nk(a) = 0. 
Thus EG x a M is elliptic with  = 0, so is formal. 



FORMALITY IN AN EQUIVARIANT SETTING 2787 

6. AN APPLICATION 

In this section we give an application of G-formality. We will shov« that the 
conqmtation of the equivariant cohomology of loop spaces simplifies considerably 
when the spa«e is G-fin'mal. 
Let us consider a silnply-commcted sImce 31. Suppose that G acts on 31 with 
non-empty fixed-point set. Let p  iii c; tre a choice of base point. Then we get an 
action of G on the loops iii I based at p, (M:p), which we shall oftell abbreviate 
as LI. Let P(ll:p) he the space of paîhs in M, based at p. Then we have the 

fitwation 

(34) 

Lll -- P(M:p) 

where rr is the lna I) sending a path "y(t) to its value at tilne 1, '3,(1). rhleover, the 
G-action induces a imll-back diagraln of fibrations 

(35) 

EG xc Lll --, EG xc P(Al:p) 
BG -- EG xo 31 

Hence there is a quasi-isomorphisln of/?('DGA's 
bv Lemlna A.a of îhe appendix. Now the inclusion of {p} into P(M:p) followed 
by w is îhe inclusioll of {p} into M. These nlaps are equivariant, so induce their 
analogs Oll the Borel constructions. Hence we get an ÇDGA quasi-isomorphism 
(37) (. «b(*),) 
Proposition 6.1. Let g act o a simply-coectcd s'pace .I with non-empty fixcd- 
point set. so thot G acts on M. Suppose tbat I is G-formal. Then there is an 
isouorph ism of R-algebras 
(38) H(.I)  TorZ,(M)(. ). 
Pro@  bave that .45(Lll ) is quasi-isomorphic to (B,.45(M),B ) (via a se- 
quence of BCDGA quasi-isomorphisms). The ssumption of G-forlnality mcans we 

have a comnmting diagram of R-algebras 
(39) R , « 
, « HS(h 
 obtain RCDGA quasi-isolnorphislnS 
(40) 

Aa(M ) « 

b(R, Ab(M),R ) ,- (R, A4¢E(IU),R)-- 
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This follows bv stalidald comparisol theorelns for the Eilenberg-Moore spectral 
sequences a.ssociated fo the bal" complexes. Thus we have that Y(R. H,,Ill). Ri 
is quasi-isolnorphic to A_;(fLI) (via a sequence of/?ÇDGA quasi-isolnorphisms). 
But the cohomology of B(R. H.(M), R) is TorH;(al)(R, R). [] 
Re'm«u'k 6.2. We can alwa..vs choose any lesolutiol fo compute Tor. But we note 
that we may always use the bar resolution, and using Lemma A.4 of the appendix. 
we see that when ;'I is G-f(»rlnal. H;(tII) mav be COlnputed via the (single) 
COlnplex 
(41) (/gR(H. H5(3I), R; 6). 
flemark 6.3. We conld also ol)tain analogous lesults fol' the equivariant cohomology 
of the free loop sl»a('e L,1l. 

ï. AN EXAMPLF 
|n this section we COlnpute ail exalnl)le of le equivariant cohomology of tle 
»ase(1-h)ol) sl)ace using fle l(n'lnalize(1 bar c«nnl)lex over  of elnark 6.2. 
7.1. Example: S 1 aeting on ILq . The circle .b ' acts on the 2-sphere .b " by 
rotations about an axis. sav the z-axis when ,5 '2 is the unit sphere in N a. This 
action is hololnOrl»hic and Halniltonian. Thus by Theorem 5.1..q.2 is G-forlnal 
(G = ,5q). It is easy to show that the equivariant coholnology ring is 
(2) , (S: »)  ['. ,]/( + 0(.- ,,), 
where the degree of .r and u is t, and R = k[u] acts as multiplication bv u. 
The fixed-point set, F, consists of the norfl and south poles, kk shall write 
F = {N,S}. Let lS u be loops based at the north pole. Then S  acts on !S . 
Then the equivariant cohomology of the based loops, Hs, (lH). lnav be computed 
as the cohomology of the bar coml)lex 
(43) ([(»[-], (.«_ ,,). []) ; 
Let w 1)e the SVlnl)lectic form on S 2. Then .r is represented - the form w,- 
uf G A.,(S2), and u is rel)l'esent('d by the forln u G A.,(S2), using the (artal 
COlnI)lex of equivariant differential forlns. Hem. f is the molnent lnap which sends 
a l)oint on S 2 C N3 to its z-colnponent. Then the inclusion of the north pole {Y} 
into S 2 induces the auglnentatiol, H.(S 2)  H}({N}) k[u,] sending x  -u 
and u  u. k omit the details of COlnputing the bar complex, but one finds 
without difficulty the cohomology generators (1. x. .... x. 1) in degree n for n odd. 
and (u /2. 1) in degree n for t even. Owing to the shuffie product structure on the 
bar complex, onc sees that. as an R-algebra. 
whel'e a'i is an indeterlnilmnt of degree i. 
Remar 7.1. In this examI)le, the lOllnalized bar COlnplex (R. H., (S). R) is 
actnallv isomOlphic to the k('DCA minimal model for £S  x s, [2j2. which is 
(45) .s, ., s = A(..,...) (& = 0: &. @ = 
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where the degrees of u and p are 2. and the degree of .r is 1. The isomorphism is 
given by (1,x, 1)  .r, (u, 1)  u. and (l.r,.r, 1)  y. 
e'ntar" 7.2. In this example, the space ES 1 xs t]S 2 is hot fornml, implying that 
tS 2 is hot G-formal. hMeed. Massey products alomM. 

APPENDIX A. [AR COMPLEXES AND EILENBERG-IIOORE THEORY 

In this aplwndix we shall discuss the theorv of Eih,nberg and Moore concerning 
tmll-backs of fibrations.  will also consider equivariant versions of these results. 
For references, sep [20], [241, or [8]. 
Let us sut)pose that we have a filnation F + E  B and a map f  X  B, so 
that we obtain a pull-back diagram: 

(46) 

Then t he maps f* and p* lnake A ° ( X ) and .4 ° (E') (difl'erent ial graded ) modules over 
.4°(/3). Let us aSSulne that B is simply-come«ed. Then a theorem of Eilenberg 
and Moore asserts that there is an isomorphism 

(47) 

0" TOl'Ao(B)(A°(X), A°(E))  H°(Ef). 

We mav use the bar resolution to obtain a resolution of. say, A°(X) bv A°(B) - 
modules. Since we are considering A°(-) to be the de Rham or Sullivan-de Rham 
complex, we will use Chen's normalized bar resolution, see [5] or [10]. 
More specifically, the bar complex is 

(48) 

X 
B(A'(X),A'(B),A'(E)) = 0 A'(X) ® (sA'(B))  ® A°(E), 

where the tensor products are over the ground field k, and s denotes the suspension 
functor on graded vector spaces that lowers the degree by one. Hence the degree 
of an e|ement (a,w ..... v,fl) is deg(a) + }-'î (deg(vi) - 1) + deg(fl), where 
c  A°(X), a:i  A°(B), and fl  A°(E). Actua||y, the bru comp|ex is bigraded. 
We introduce the bar degree, denoted B(A°(X), A°(B), A°(E))o. The bar degree 
of an element (a,a: ..... a,',/3) is defined to be -k. The other grading is the 
normal tenso," product grading, the degree of ail element (ca. ' ..... '., d) heing 
k 
deg(a) + i= deg(cvi) + deg(fl). 
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There are two differentials 
(49) d(a. c0,..., c0k, 3) = 

+ 
(.50) -ri(a. ..... ».) = 

of total degree +1: 
(c/a. 1 .... , k,/) 
/,. 
+ OE(- 1)ei-'+l (0¢. c01 ..... cOi-l.d:i.oei+l ..... 
(-)(a.«. .,. 
k-1 
+ (-)«-'+(. ..... 

where ei = degc + (legw +---+ degwi - i. The differential 5 has degree +1 with 
respect to the lal grading, while the differential d bas degree +1 with respect to the 
tensor produçt grading. ()ne mav verifv that da + ad = 0. and we put D de___f d -F ( 
to be the total differential. With the given bigradilg, we get a double complex 
with the two difl'erelitials d aml ri, which giw,s rise to the Eilenberg-Ikloore spectral 
sequence. 
('hen's norlnalized version of this bar eoln|lex is the following. If f ff 4°(B), let 
,5'i(f) l,e the opelator on B(A'(X). 4"(B), .t'(E)) defined bv 
(51) Si(f)(a. COl ..... la)k,/ ) ---- (tq, kU 1 ..... kUi__ 1. f., ..... 
for 1 <_ i _< /« + 1. Let II be the subspace of B(A'(X), A'(B), A'(E)) generated 
bv the images of Si(f) and DSi(f) - Si(f)D. Then define 
(52) (A'(X),A'(B),A'(E)) dej B(A'(.\').A'(B),A'(E))/II. 
Then I| is closed under D, and when H°(B) = I« (B is connected), then I1- is 
acyclic, so that /)(A'(X), 4"(B), A'(E)) is quasi-isomorIhic to B(A'(X), A'(B). 
A'(E)). Notice that in the normalized bar con@ex there are no elements of neg- 
ative degree, and with out assumption that /3 is simply-commcted, we are assured 
convergence of the associated Eilenberg-hloore spectral sequence. The map 0 men- 
tioned above is induced by the ma l) 
(53) 0- B(A'(X), A'(B),A'(E))--, A'(E/), 
which sends all tensor products t o zero except fol" A ° (X)®k A ° (E), where the map 
is (a. [3) /5"0 A f*3. Note that 0(lI') = 0. so that we get an induced lnap 
(54) 0"/(A°(X), A°(B). 4°(E))-- A°(E.). 
The nornlalized bar complex may also be auglnented. The augmentation, e, 
nlai)s all elements of positive total degree to zero. The elements of degree zero 
have the form (f. g), where f ¢ 4°(X) and g ¢ A°(E). Then we define e(f,g) = 
ex(f)eoe(g) = f(xo)g(eo), where x0 and e0 are chosen base points in X and E, 
respectively, and ex,ce are the augnmntations of 4°(X), A°(E), respectively. If 
we choose base points so that the pull-back diagram above preserves all base points. 
then 0 is an augmentation-preserving map. 
The bar conlplex has a natural coalgebra structure. Since we are inputting 
kCDGA's to the bar complex, we also obtain a structure of kCDGA on the bar 
complex via the shuffte product. 
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More specifically, if (et 1 ..... (gp) and (bi,..., bq) are two ordered sets, then a shuf- 
fie r of (al ..... Op) with (bi ..... bq) is a pernmtation of the ordered set (al ..... 
bi,..., bq) that preserves the order of the ai's as well as the order of the ba-'s. That 
is, we delnand that if i < j, then a(ai) < a(aj) a,,a «(bi) < «(b¢). 
We obtain a product Oll B{A'(X), A°(B),A°(E)) by first taking the normal 
tensor product on the A ° (X)® A ° (E) factors, then taking the tensor product of this 
product with t.he shufl3e product on the A°(B) ®i factors. As usual, we introduce 
a sign (--1) deg(a)deg(/) whenever o is moved past/3. One che«ks that this product 
induces a product ol, Chen's norumlized con,plex,/)(A" (X),. 1 ° (B), A" (E)). as well. 
Thus we arrive a, t|le following lmnma, whosc proof is left to the reader. For more 
details, see [16]. 

Lemma A.1. A.'surne that we bave the pull-back diagvam (46), wh«re p is a fibva- 
tzon and 13 is sirnply-connected. Then the nornalized bar «omple.c 

is a kCDGA. Moveover, 

0 : O(A'(X). A'(B), A'(E)) -- 

is a quasi-isomorphism of kCDGA "s. 

Remark A.2. We note that Chcn's normalization is flmctorial. That is. if we haxe 

a conmmtative diagram of kCDGA's 

then we get a map of kCDGA's/)(A, B1,C1) --,/)(Ae. B2,C2). 

We may fornmlate an equivariant analog of the bar complex. Let us consider 
again the pull-back diagram (46). If we suppose flrther that X.B. and E are G- 
spaces, and that f and p are equivariant maps. then we obtain a pull-back diagram 

(56) 

EGxX ï , EGxB 

Note that we are assuming B to be simply-connected, which in turn implies that 
EG xo B is simply-connected as well. We ma- apply Lemma A.1 to the diagrmn 
(56). However, the bar complex (A(X), A(B), Ab(E)) has the extra structure 
of an RCDGA ° or RCDGA, depending on fixed points. We mav give it an R- 
algebra strucure via the R-algebra structure on the Ab(X ) factor, and we define the 
attgmentation as above, assmning that we tan choose our base points as described 
before to be actually fixed points of the group action. We arrive at the following. 
Lemma A.3. Assume that in the pull-back diagram (46), we have that X, B. and 
E are ail G-spaces wilh f andp equivariant maps. Then the normalized bar complex 
(A(.'(),.4c,(),.%(E)) 



2792 STEVEN LILLYVHITE 

is an RCDGA °. Movcover, 

is a quasi-isomorpbism of RCDGA °'s. If we asu'me further that all fixed-poznt 
.ets are "on-e'mpty. and the diagram (46) preserves base pohts chosen from the 
various fixcd-pont sct., the te normdizcd bar complex is at RCDGA. and 0 s 
a qua.i-isonorphism of RCDGA "s. 

In this equivariant case, we lnay flnthcr simplify the bar COlnplex, following an 
idca of [10]. Let us consider the bar COlnplex over R: 

(57) 

s, (A5EE (X), AS(S),-«5E(ï)) = ( A5EE(X) c,. (.-5 (S))® «) 5(:). 
t-0 

where ail thc tensor products are over 

Lemlna A.4. Sqqose tbat ,4.13. avd C are RCDGA "s and we bave .morpbisms 
of R('DGA "s ,4 -- B - C, wbeve R = H°(BG) for (7, a compact, connected Lie 
gvow. (II'e use this scquctce to defiw a (differetial gvaded) B-module structure 
o' A ad C) &ppose, flwtber, eitbev tbat for each r  R. r is ot a zevo-divisor 
ir A. or tbat this conditio't bolds for C. Tbev tbe natural projection 

(58) 

t.(A, t. C)  t.A, g, c) 

is a quasi-isomorpbisrn of RCDGA 's. 

Pro@ We have that Bn(A. 13, C) = Ba(A. 13. C)/V, where 1" is the sub-complex 
generated by all elements of the form 

(59) 

(a. bi ..... rb ..... bk, C) -- (a. bi,..., rb+ ..... ba, c). 

where r  R,a  A,b i  13. and c  C. It is due to the fact that all elements of R 
bave even degree that l: is closed under the differential D = d + 5. We clailn that 
V is, iii fact, acyclic. To see this. consider the lnap s  l "i - l-i-1 defined bv 

(60) s{(o,b I ..... ïl)i,... ,bh.c) -- (a. bl .... ,'rbi+ .... ,ba.c)} 
= (-1)  {(a, bi ..... rbi. 1. bi+l,..., bk, c) - (a. bi, .... bi, r, bi+l ..... bk, c)}, 

where ai = deg a + deg w +-   + deg cri - i. It is straight forward but t edious to check 
that ris + sd = 0. and that 5s + s5 = id., so that Ds + sD = id., and consequently 
V is acyclic. Moreover. it is easy to check that l" is an ideal, so that the product 
on thc bar COlnplex induces a product 011 the bar complex over R. [] 

Remark A.5. Lemlna A.4 is valid using the normalized bar complex. 

Corollary A.6. In the situation of Lemma .4.3. 

(6) 

O'n(A.(X),A.(B),A.(E))--, Ab(E.f ) 

is a quasi-isomorpbisn o.f RCDGA  "s (RCDGA "s). 
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LARGE RECTANGULAR SEMIGROUPS 
IN STONE-(ECH COMPACTIFICATIONS 

NEIL HINDMAN, DONA STRAUSS, AND YEVHEN ZELENYUK 

ABSTRACT. VP show that large rectangular semigroups caI he round in certain 
Stone-Cech compactifications. In particular, there are copies of the 2 « x 2 « 
rectangular semigroup in the smallest idem of (fiN, +), and so, a semigroup 
consisting of idempotents can be embedded in the smallest idem of (fiN, +) 
if and only if it is a subsemigroup of the 2 ¢ x 2 ç rectangular semigroup In 
fact, we show that for an 3" ordinal ,k with cardina.lity at most ¢, fin contains a 
semigroup of idempotents whose rectangular components are ail copies of the 
2 « x 2 ¢ rectangular semigroup and form a decreasing chain indexed by ,k + 1, 
with the minimmn component contained in the smallest ideal of fiN. 
As a fortuitous corollary we obtain the fact that there are <L-chains of 
idempotents of length ¢ in fiN. \Ve show also that there are copies of the 
direct product of the 2 c × 2 c rectangular semigroup with the free group on 2 ç 
generators contained in the smallest idem of fiN. 

1. INTRODUCTION 
The Stone-Çech «ompactification of the integers fin has a semigroup structure 
which extends addition on N a.nd has significant applications in Ramsey Theory 
and topological dynamics. Some questions about the algebra of fiN, which sound 
deceptively simple, have been found to be extremelv ditïtïcult. For example, it 
is hot known whether/3N contains any finite semigroups whose members are hot 
all idempotent. VChether there were two idelnpotents in /31%1 whose sure was an 
idempotent different Kom either remained an open question for several vears. It 
was answered in the aflïrmative in [101, in which it was shown that a certain finite 
rectangular semigroup could be embedded in/31%1. (A semigroup is rectangular if 
and onlv if it is isomorphic to the direct product of a left zero semigroup and a 
right zero semigroup. A rectangular compo'net of a semigroup of idempotent, is a 
nmximal rectangular subsemigroup. As suggested bv the naine, distinct components 
are disjoint. The components are partially ordered bv the relation /9 _< Q if and 
only if/9 0 C /9, equivalently 0/9 Ç /9 [7. Theorem 1].) In this paper, we show 
that the rectangular semigroup 2 « x 2  with the first factor being left zero and 
the second right zero. can be embedded in /31%1. Indeed. pin contains semigroups 
of idempotents which are the union of ¢ rectangulai components each isomorphic 
to 2 « x 2 «. We shall show also that if S is an infinite cancellative semigroup with 
cardinalitv e:, then S \ S contains a semigroup of idempotents which is the UlliOll 
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of at least t rectangular COml)onents, each isomorphic to 2 2" x 2 , with the first 
factor being lefl, zero and the second right zero. 
X first review terminology used in the topological theory of semigroups. Let 
,q' 1)e a semigr¢mp and a topological space. For each s  .', we define mappings 
A. and p. from S to itself l»v A,(/) = .«t and p(t) = ts. ,ç is said to be a right 
lopologicol semigroup if p. is contimous for every s  S. In this case, the topological 
tenter A(S) of S is defined by A(S) = {s  S  A is continu(nlS}. S is said to 
l)e a ,semitot)ological semigrou I) if ils and As m'e hoth contimous %r every s G ,ç. 
It is sai(l te) t)e a topologicol scmigroup if the semigroup Ol)eration is a continuous 
lnal)ping from .%' X ,5' to 
If ,S' is a discrete semigronp, we regard its Stone-'ech «ompa«tification 5' as the 
spaçc of ultrafilters defined on ,5 . with the topology defined bv choosing the sers of 
the fonn 4 = {p G/35  .4 G 1»} as a baso for the open sers. t3S is then a compact 
tIausdorff simce and . = ('ÇBs(4). XX]' regard S as a subset of 3S, b identifying 
oach clé,mont of .h' with the principal ultrafiltor that it defines. N can he given a 
scmigroup structure which extonds the scmigronp structure of S in snch a way that 
 b' is a compact right topologiçal scmigronp, with ,S" contained in its topological 
('enter. If A ç S..I* will denotc A  A. 
XX3  shall nso basiç algehraiç 1)r»i)crtios that hold in all compact Hansdorff right 
topologi«al scmigroups. (XX shall ho assmning that ail hypothesized topological 
spaces are ttausdorff.) A simple and important property is that everv compact 
right topological semigroup ç contains an idemt)otent. T has a smallest ideal 
K(T), which is hoth the union of ail the lnillimal left ideals and the union of all 
the nlillinlal right ideals of T. Every right idem of T contains a lnillilnal right ideal. 
and every lefl ideal of T contains a mininlal left ideal. If L is a minimal lefl ideal 
and B a lnininlal right ideal in T. tllen RL = R  L is a group. So B  L contains a 
unique idempotent. If f  T  T' is a honlolnorphisni ff'oin T onto a COlnpact right 
topologi«al grou I) T'. thon f[K(T)] = K(T'). For each minimal right ideal R' of 
T', there is a minimal right ideal R of T for which f[R] = R'. The corresponding 
statement holds for left ideals as well. There are three natural orderings of the 
idempotents of T defined 

C<_Lf <= e = cf , 
e<_Rf  e= fa. and 
c<_f  of=fa=c. 

An idempotent e is minimal with respect to any or ail of these orderings if and onlv 
if e e K(T). The reader is referrcd to [1], [6], or [9] for proofs of these statements. 
When S is a discrete semigroup, the smallest ideal K(/3S) is of special importance 
for combinatorial applications, and in particular, the members of idempotents in 
K(/3S) have strong combinatorial properties. (Sec [6, Chapter 14].) Thus we are 
especially interested in those scmigroups of idempotents that tan be embedded in 
the smallcst ideal of/_S. 
As we have already mentioned, a semigroup S is rectangular provided it is iso- 
morphic to the direct product of a left zero semigroup with a right zero semigroup. 
This is equivalcnt to saying that it satisfies the identities .r 2 = x and .ryz = xz. 
(The necessity is trivial. For the sufficiency, pick x  S, note that Sx is a lefl 
zero scnligroup, xS is a right zero semigroup, and the fimction (a. b) - ab from 
S.r x xS to ,_q is ml isonlorphisnl.) We observe that a rectangular semigroup S 
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satisfies S = N(S) = LR  L x R. where L deuotes anv minimal lefl idem and R 
any nlinilnal right idem in S. 
If S is a senligroup, E(S) will denote the set of idempotents iu S. 
If A is anv set, 72I(A ) will denote the set of finite nonempty subsets of A. 

'2. TtlE SEMIGIOItPS 
The subsemigronp ]HI =  cg(N2 '') of (tiN, +) hokls ail of the idempotents 
 L 
of ÇN and lllllc of the klloWll algebraic structure. (Sec [6, Section 6.1].) It oecllrs 
widelv in the stndv of semigroups of the fi»rm /3,% If S is an infinite discrete 
cancellative semigroup, every G6 subset of S* that «ontains an idempotent, contains 
copies of N [6, Theoreln 6.32]. N also bas the property that anv compact right 
topological semigroup with countable dense topological eenter is the ilnage of 
uuder a contimmus honomorphism [6. Theorem 6.4]. In this section we introduce a 
senligroup . whi«h satisfies a similar conclusion fin" an arbitrarv infinite cardinal 
. As a consequence of the results of thc next section we shall conchule that ea«h 
N. contains large re«tangular subsemigroups. 
Definition 2.1. Let h be an infinite cardinal. Thon 
supp(z) : {t < n " .ca ¢ 0}. For et < n, eu is that nlember of II such that 
supp(ea) = { a }, and 
The structure of  is that induced bv an "'oid'" as imroduçed bv John Pym [8]. 
When we sav that two structures are "topologically and algebraically isomorphic". 
we nlean tllat there is one fim«tion hetween them that is both an isomorphism and 
a honeomorphism. 
Theorem 2.2. The compact right topological semigroups  attd  are topologi- 
call9 and algebraicall isomorphic. 
Pro@ [6, Theorem 6.15]. 

It is a fact [6. Lemma 6.8] that ail of the idempotents of J3N are in IHI. Thus, bv 
[6, Theorem 1.651, Iç(]HI)= Iç(3N) a IHI. 

Theorem 2.3. Let S be a countablg mfinite discrete gT"oup. Then ris \ S contains 
a topologi«al and algebrai« copy T of IHI such that Iç(T) = Iç(/3S) ç T. 

Pro@ Take any idempotent p C K(/3S). By [6, Theorem 9.13], there is a left invari- 
ant zero-dimensional Hausdorff t.opology on S in which the ultrafilter p converges 
to 1. Then by [6. Theorem 7.24], with X = G = V(a) = S for everv a C G, there 
is a topological and algebraic embedding f : IHI --* ris \ b' such that p  f[IHI]. It 
remains to apply [6, Theorem 1.65]. [] 

A similar result applies to the senligroup (N,-). Given n G w we define the binary 
support of n by n = Etesupp2(n ) 2 t and supp2(O 

Theorem 2.4. Let S = (N, .). There is a topological and algebraic copg T of 
contained in /3S\S which contains all of the idempotents of S\S. In particular. 
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Pro@ Let (pi)ici__ 1 be the sequen«e of primes. Then (Oi-----1 03, --) is isolnorphic to 
(N, .) via the lllftlï) 
let tri : S  co be the projection to the ith factor and let i : /3S -- /3o be its 
contilmous extension. 
Let {Xi : i  N} be a partition of o into infilfite sers and for each i  IN, let 
c)i : Xi -- o be a bijection. We define 0 : co -- S by agreeing that for each i  N 
and each n  

where ïje 2 e'(j) = 0. k ' note that 0 is a bijection. (If j  supp(n) supp(m), 
then for Solne i, j G Xi and so i(j)  sui,P2 ((n)) supp 2 (i(m)). Also, given 
z  S. fbr each i  N let  = i-l[supp (i(x))], let Z = ie }, and let 
' = tez 2t" Then 0(n) = .r.) Colsequently by [6. Exercise 3.4.1] the continuons 
extension     S of 0 is a bijection. SillCe 0(n + m) = 0(n) +0(m) if 
supp2(n )  supl)2(Ttt) = ,  is a holnOlnorphism on  1,y [6, Lelmna 6.3]. 
lb çolnplete the proof, let p be an idempotent iii SS. Since 
a[NI= [N,=ll{0}: N,,=0[]k{0}. 
it suces to show that for all n  N, 0[2"]  p. So let n  N and suppose that 
0[w2 ' ]  p. Pick t e {1,2 ..... 2'- 1} su«h that 0[w2  + t] e p. pick j e supp2(t), 
and pick i such that j  Xi. Now Yi(P) is an idempotent: so either (p) = 0 
or (p)  flNN. Tlms by [6, Lelnlna 6.6] w2 e'(j)+l  p. Pick z  w2 ()+1 N 
i[O[2'+t]] and pi«k k e 2"+t such that .r= i(O(k)). Then j e supp2(k)NXi; 
so i(j)  stt)P2(x), contradicting the fact that z  2 (j)+l. 
Observe that if  > w, then N N K(ll ) = . To see this, one lets p  N and 
q e Nk<w d {e IlÇ{}  nlill sllpp(w) 2  and lllax supp(z) < w'}. 
Then p  I1  + q + p and so [6, Theoreln 4.39] applies. 
Theorem 2.5. Let  be an inflnite cardinal and let T be a compact right topological 
semigro,p. Assume that there is a set A ç A(T) s,ch that lAI   and A is dense 
in T. Then there is a contim, o,s s,wjective homomorphism f " N  T. 
Pro@ Elmlnerale A as {t : a < }. with repetition if lAI < . Let {I :  < } 
be a partition of  into subsets of size . Define h  IV + T by first agreeing 

that for each o. < t;, h(e,) = tx, where o.  I. 
h(_.a F ea) = HaF h(ea), where the product is 
indices. Define h(0) arbitrarily. Let ï :/3II, -- T be 
and let f be the restriction of  to IE,. 

Then for F  7)(t), define 
taken iii increasing order of 
the contimous extension of h 

To see that h [IE,] = T, it suffices to show that A C_ h[lE,]. Given "7 < t, we have 
that II[ = t. Pick a t-uniform ultrafilter p on {ec  a  I-}. Then p  IE, and 
h(p) = tv because f is constantly equal to t- on {ec  a  L}. 
To see that f is a homolnorphisln it suffices by [6, Theorem 4.21] to observe that 
whenever a:  IIÇ\{} and /  IlÇ\{} with rein supp(/) > max supp(x), then 
(z + v) = h(z). (v). [] 

Definition 2.6. Let S be a selnigroup, let h; be a cardinal, and let (t,x),x< be a 
-sequence iii S. 
(a) Given F  7)l(n ). 1-leF t is the product in increasing order of indices. 
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(b) If D C_ h;, then FP((t)aD) = {IIF ta  F E PI(D)}. 
(c) The sequcnce (ta)a< has distinct finite prodacts if and onlv if whenever 
N. G E PI(A) and 1-[,XeF t,x = 1-[aec fa, one must bave F = G. 

Theorem 2.7. Let S be an infinite cacdlative discrete se'migroup with «ardmality 
ris \ S coutoins a topological ad algebraic copy of ,. 

Pro@ By [6, Lemma 6.31], we nlay choose a t-sequen('e (ta)a<,, in S with distinct 
finite products. Let T = FP((t)x<). For each V < , let Tv = FP((t),<x<). 
U put ç = < «s(T). By [6. Theorem 4.20],  is a subsemigroup of riS. 
 define 0  T  I1 by 0(xe  tx) = xe «» (Since the sequence (t)a< 
has distinct finite products, the fimction 0 is well defined.) Let  clos(T)  flllç 
denote the continuous extension of 0. By [6, Theorem 4.21], the restriction of g to 
ç is a homomorphism. Nov  is injective, by [6. Eercise 3.4.1]. Since for each 
7 < , [T] = cft3u-{x e II/  lnil SUl)p(x) > 7},  maps  onto N. Thus  
determines ail isomori)hisnl from  OllIO .  

3. CHAINS OF RE('TANGI'LAR SEMI(IROI'PS 1N - 
Let  be an infinite cardinal and let IÇ denote the rcctangular semigroup 
with the first factor 1)eing left zero and the second right zero. We show in this 
section (in Crollary 3.10) that for mly infinite cardinal h;. algebraic copies of 
can be round in tç(NI,). Indeed. if A is any ordinal for which I)q _< , there is 
a (lecreasing chain (D»)»<a of diioint copies of I.), contained in NI, with Da 
embedded in tç(NI,). 
Notice that 1/ contains a copv of cvery l'ectangular semigroup of cardinalitv at 
lllOSt 

Definition 3.1. Let A be any ordinal and let A be all.V llOllelllpt.v set. Let 0 denote 
a selected elelllent of A. For p < A. let Cp = A x 4 x {p} and let 
C = Ca,a = A U Up<aCp = A U (.4 x A x A). 
The operation  on C is defined as follows. Let a, b. c, d E .4 and let p. q < A. Then 

a.b = b, 
o. (b,c,p) = (b.c,p), 
(b,c,p).a = (b,a.p), 
(a,b,p).(c,d,q) = (a,d, pVq), 
where p V q is the maximum of p and q. 

V'e leave to the reader the routine verification that the operation on CA.a is 
associative. Notice that for any p < A, Cp is a cop.v of l/iA I. 
Definition 3.2. Let A be an ordinal and let p < A.  let vp = (0,0.p), and 
for every x = (a, b.p) e C, we let xt = (a. 0.0) and Xr = b. For x  A, we let 
The folloxving is silnple, and its proof is like that of [6, Theol'en 1.46]. 
Lemma 3.3. Let S be a semigroup, let H be an ideal of S. let L be a minimal 
l@ ideal of H. let  be a minimal right ideal of H, and let z E S. Then Lx is a 
minimal l@ ideal of H. .rR is a miimal right ideal of H. xL ç L. and Rx ç . 



2800 NEIL HINDMAN, DONA STRAUSS, AND YEVHEN ZELENYUK 

Pro@ It suffices to establish the assertions about La" and 3"L. Now L.r C H.r C H 
and HLx C L.r: so Lx is a left kleal of H. Let M be a lcft kleal of H with M ç Lx. 
Let.l={gL'gxM}. GivcngJandzH, wehavezyLand zyx3l: 
so zg Ç J. Thus .] = L and so M = L.«. 
Next. givcn g G L = Hg, so pick z Ç H such that y= zy. Then xy=xzy Ç 
H9 = L.  

Lemma 3.4. Let A bc o nonempty set with di.çling«ished element O. and let C = 
C.4,1. Let T bc a right lopological semigroup, and let f  T  C be a surjective 
homomorphism for u,hich f-[A] and f-l[Co] are conpact. Then there is a homo- 
morphism g C  T sach that f o g is the idcnlity on C and 9[Co] ç Iç(f-[Co]). 
(f T is compoct, then 9[Co] ç K(T). 

Ptv@ V\, first (lvfine g on .4. We have that f-1 [.4] i8 a COlnpa('t semigroup. Choose 
a minimal right idem Nof f-[A]. For ea«h o  A. f-il{c,}] i . ,,f id(, of f-l[A]. 
So (.hoose a llininal l(,ft id«al « of f-l[A] with S ç f-[{}], ald let g(a) be the 
identity of the gloup N  a- Then immcdiately f(g(a)) = a. Also. given a, b Ç A 
we have that .q(a) and g(b) are idt'mpotents in N: so g(a)g(b) = g(b) = g(ab). 
Let = {(,.o.o):o  4}. Tll('ll  is a l('ft idçal of Ç0: so f-l[B isa left ideal 
,,f f-a[c0] whi«h thelef(n'e contains a minilnal left ideal L of f-l[C0]. For each 
a G A let F = {(o. b, 0) : b G A}. Then F is a right ideal of C0. So pick a lnininml 
right ideal R of f-[C0] with R ç f-lirai. Bv LCIlIIlI& 3.3, SillC f-l[C0] is ail 
i(leal of f-l[A uÇ0], we bave that 9(0)- R is a nfiniliml right ideal of/-l[C0] and 
L. 9(a) is a nlinilnal lcft ideal of f-z[C0]. For a. b G .4. let 9(a, b. 0) be the identity 
of the group 9(0)" R. L.9(b). Notice that if T is COlnpact. then K(T) ç f-[C0] so 
that Iç(f-l[C0])  Iç(T) and thus 9(a, b, O)  K(r). Also 9(a, b. O) = 9(O).x.-g(b) 
for SOllle .l'e a alld SOllle  e L. So f(.q(a,b.O)) = O. f(x). f(y).b= (a.b, 0). 
To conclu(le the proof we nee(t to show that g is a hOlnOlnorphisln. First we let 
o. b, c Ç A and show that g(o) .g(b. c. O) = g(b. c. 0) and g(b. c. O) .a = g(b. a. 0). Pick 
.r  H6  L such that g(b. c, O) = 9(0)  x. g(c). Then 

g(o).g(b,«.O) 

= (,)-9(0)-.,-9() 
= 9(0)- .- 
= 9(b, «, 0). 

So the first clainl holds directlv. Multil)lying on the left by g(b.c. 0) and on the 
right by g(a) Olm sees that g(b. c. 0)-g(a) is idelnl)otent. Since g(b.c. 0). g(a)  
9(0)" Rb" L. 9(c). g(a) = 9(0). Rb" L. 9(a), we nmst bave that g(b. c, 0). g(a) is the 
identity of g(0)  Rb" L. 9(a), nalnely 9(b, a, 0). 
Finally, let a, b. c, d Ç 4. Then 9(a. b. O)  9(c, d, O) Ç 9(0)  R,  L  g(b)  9(0)  
R«.L.g(d) c 9(O)-R«-L'g(d). So it suffices to show that g(a.b.O).g(c,d,O) 
is idempotent. These elements satisfy 9(a.O.O)  L  g(O) and 9(c, 0,0)  L  
9(0). So as idelnpotents in the saine lninimal left ideal of f-[Co], we bave that 
g(a, O, O)  9(c, 0.0) = 9(a. 0.0). Recall that we bave shown that for any x. y  A. 
9(0)-g(x,y,O) = g(x,y,O) and g(.T.y,O)-g(O) = g(x,O.O). Thus we have 

g(a, b, 0). g(c. d. 0). g(a, b. O) 

= 9(a,b.O).g(O)-g(c,d,O).g(O).g(a.b.O) 
= g(a,O.O).g(c.O.O).g(o,b.O) 
= g(o,O.O).g(a,b,O) 
= g(o,b.O).g(O).g(a.b.O) 
= 9(a,b.O). g(a.b,O) = g(a,b, 0). 



IECTANGULAIR SEMIGROUPS IN STONE-CECH COMPACTIFICATIONS 2801 

Multiplying on the right by g(c, d, 0) we have that g(tt, b. 0).g(c, d, 0) is idempotcnt. 

We now cousider the situatiou in whi«h k > 1. For k > c0 we do uot necessarily 
get that g is a homomorphism, hut xve corne 

Theorem 3.5. Let A be a nonenq#.q set u,ith distinguished element O. let A be an 
ordinal, and let C = Ç4.. Let T be a right topological semigmu p, and f : T 
be a su«jectiue t, omomo,vli.m .sud, tlmt f-[A] i.s. compact and f-l[c] is comp,wt 
for every p < A. The'n there is a function g : C  T su«h th«.t f o g is the identity 
attd 9 bas the .followin 9 pvperties: 
(i) If q  p < .  G Cp, and g G AUC. then 9(«g) = g(a')'9(g) and g(g).g(.r) 
is art idempotent in thc saint minimal lefl id«al 4 f-'[G] C,,s 9(.qx). 
(ii) lf p < A. « G C v. and U G 4  Co, then 9(g) " 9(.v) = 9(g.r). 
(iii) ff q  p < A. n G . p = q+n. g G C. and x 
(iv) gp < A. ther, 9Cp]  lç(f-l[cv]). 
The seigrou p T «ontins  semigrou p D = p«A Dp of idempotents where ea«h 
is a ,ecta,gthtr component of D with g[Cp] Ç Dp and the scquence {Dp)p< is de- 
cre«sing in the orderin 9 of components, so that for each p < 
6lAI  IX]  w. then for et, ch p < X. Dp] i.ç i.çomo,phi« fo I]A I. 
Proof For p < X we &'fine g on A U qp Cq 10 induction on p so that g satisfies 
conclusions (i), (ii), (iii), and (iv). By Lemma 3. we mav definc 9 on A U C0 so 
that g satisfies (iv) and is a homomorphism and therefiwe satisfies (i). (il), and (iii). 
Now let (l < p < X and assume that 9 has been defined on 4 U q<p Cq. 
IO show fiist that we moEv choose a minimal left idem L of f-[Cp] such that 
L ç q<vf-l[Cp].g(%)and fiL] = Cp-%. A si,uple computation establishes that 

x. 9(q) 

= .. g(), g(,,,.) 
 f-'[%]. (,,). 

by (i) 

Consequently. r-]q<p f-'[Cpl- g(uq) is a lefe ideal of f-l[Cp] mM thus contains a 
mi,,i,nal left idem L of 
Now giveu .r  L, one bas x  f-a[G] g(uo). So for sou,e «,.  A. f(x) = 
(a,O,p)  G- up. Thus f[L] 
If p is a successor ordinal, observe that g(%_).f-[Cv] is a right id « f-[Gl 
and pick a minimal right ida 
is a minimal right ida « G ad f[l ç %-" G: o f[l = -1" G- 
If p is a limit ordinal, note that f-[%- Cp] is a right idem of f-[cp]. So pick 
a minimal right ideal 
Now f-[cp] is an idem of f-[AuCUCp]. So by Lemma 3.3. for any x  Cp, 
g(xt) - R is a minimal right idem of f- [%] and L-g(r) is a minimal lefl idem of 
f-l[c A. çhr«, (Ot)" 
g(a') be the identitv of 0(.re)  R- L- O(a>). Notice that (iv) is satisfied. 
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To verify (i), let q < p. let x  Cp, and let y  A tO Cq. To see that g(xy) = g(x). 
g(y}, we show that g(x}  g(g) is an idempotent in the saine grouI) as g(xy). Since 
(J'g)e = xe and by Lemma 3.3, R. g(g) Ç R. we have that g(xg)  g((xge)) " R = 
(-«)  n ,« g(,). ()  (e)  n-() ç (e)  n. 
also. (a.g},. = y and so g(a'g)  L.g((xy}) = L.g(y). ço see that g(x).g(y) e 
L. g(y), we consider two cases. If g e Q, then g(x)  g(g) e g(x)  L. g(Yr) ç 
L" g(y,.). Now assmne that q < p (and y  A U Cq). Note that L ç f-[Cp]. 
g(«q); so g(uq) is a right identity for L and thus L : L. g(uq). Also. a simple 
computation establishes that uuy : uqy. Therefore, using the fact that (i) holds 
at q, g(uq) . g(x) . g(y) : g(«qx) . g(y) = g(t, qxy) = g(uqy) : g(uq) . g(y) and 
th, g().9()  r.9().9(y)= r-9(,)-9(«)-9(Y)= r'9(,)9(>)= 'g(>)- 
(.onsequ¢ ntlv, we have in anv event that g(xy) and g(x). g(g) are members of the 
group g(xt)  R . L. g(y).  show that they are equal bv showing that g(x)  g(y) 
is idempotent. 
Since g(x), g(0) : g(.r,.0) = g({}) : g(g0) = g(y)  g(O) we have that g(x) - 
g(O)  L. g(x)  g(0) = L-g(0) and be«ause g(x)  g(y)  L. g(y) we bave that 
g(«). g(). 9(0)  . g>). 9(0) = L. g(0). 
w g() = 9(.,c). -9() o- ,, z  -n..o 9(0)- 9(*e) = g(0e) = 9(xe) 
and so g(0)  g(x) : g(xe) . z. g(x) : g(x). If y G Cpwe bave similarly that 
g(O). g(y) : g(y), while otherwise g(0)- g(y) : g(Oy) : g(y) by (ii) of the induction 
hypothesis.  have that g(x)-g(0)-g(x), g(O) : g(x) . g(x) . g(O) = g(x) . g(O). 
So g(x)-g(0) is an idempotent in L. g(O), which is a minimal left ideal of f-l[cp 
by Lenmm 3.3. Therefore g(x)  g(0) is a right identity for L  g(0) and thus g(x)  
9(). g(0). g(z). g(0) = 9(-). 9(). 9(0). So 

g(x). g(y). g(x). (y) 

= v(,). v(y). g(0). v(,), g(0). 
= g(x). v(). v(0). v(v) 
= (,). g(y)-(y) = («). 

as required. 
Bv Lemma a.a. oy).g(xe). R ç f-l[cp]; so (y)'g(x) E 
L-g(«,.). Also, g(yx) e L. g((gx),) = L. g(x,) and by Lemma a.a. 
is a minimal left ideal of f-l[Q,]. To see that g(y)  g(x) is idempotent, note 
that xyx = x. So g(x). g(y). g(x) = g(xg)- g(x) = g(.'gx) = g(x) and t5us 
g(g) . g(x) . g(y) . g(x} = g(g) . g(x), as required. This completes the verification of 
(i). 
Toverify (il), let. x C, and let y  AtoC0. Pick z L-Rsuch that g{x) = 
g(xe)  z-g(x,). Then g(y)  g(x«) = g(yxe). If y  -1, then yxe = xe so that 
g(y).g(x) = g(y).g(xg).z.g(x,) = g(xg).z-g(x,) = g(:r} = g(yx). So assume that 
g E C0. Then yxe = (yx)e and (y«), = x,. So g(y). g(x) = g(y). g(xe)" z. g(x,,) = 
.q(yxe). z.g(x,) --- g((yx)t), z.g( (yx),). So to see tiret g(y).g(x) = g(yx) it suffices 
to recall ïrom (i) that g(y)  g(x) is idempotent. 
To veriïv (iii), let n  w and let q <_ p such that p = q + n. Let x  Cp and let 
g ff Cv. If n = 0, the conclusion ïollows ïrom (i). So assume that t > 0 so that p is a 
suc«essor ordinal and p- 1 >_ q. Now (ya')e = ye: so g(yx) ff g((yx)g). R = g(gê).R. 
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Recall/hat R C_ 9(up-)  f-x[C v] and conse<lUently R = 9(up-,)" . Thus, 
9(9)-9()  9()9(,r«)-  
= g()'g(,r«)'g(llp_l)" 
= g(9.rc)-g(v-)" R by (i) at q 
= g(,rftt, p_l). R by (iii) a.t p- l 
= 9(yeup-)" R 
= g(Ye) "g(up-1)  R by (ii) at p- 1 
= g(e) . 
Now 9(Y)" 9(x) is an idempotcnt in thc sa.me minimal left ideal of f-[Cp] by (i). 
Since g(yx) and 9(Y) " g(x) arc also in thc saine minimal right ideal g(y()  t. they 
must be equal. This complotes the induction step. 
Next, we establish (v). So assume that T is compact and A is a successor. Thon 
Iç(T) Ç f-l[C_l] and so h-(f-l'[C_]) ç h'(T). Sin«e g(x) e lÇ(f-l[CA_l]) for 
every x  C-l, (v) holds. 
5r each p < A, let 
 = { (')  F e ç«(a), » = ,,,x F, ,,a fr «h q e F, ' e C}, 
where for each F, thç product qF g(,rq) is taken in increasing order of indices. 
 show now bv induction on IF] that 
ifp < A. y  Cp. F  çy(A), xq  Cq for each q  F, 
(*) and maxF  p. tlwn g()'qF g(q) = g(Y'qF q)" 
Let r = maxF. If F = {r}, then g(y)-g(x) = g(yx) by (i). So assume that 
() « (.) = () a (Æ)a() 
= 9(Y" ea xq)  g(x.r) by the induçtiou hypothesis 
= a(a r) b (i). 
Now we show that each member of D is idempotent. So let F  çy(A). let 
p = nmx F, and for each q  F, let xq  Cq. If F = {p}. then g(Xp) is idempotent. 
So assume that IFI > 1 and let a = F  {p}. Then 
« a() « () = a a(«) a(«) « () 
= a («) 
= a ()a( 
= a ()a() 
= qF g(.rq) . 
Now let r.p < A, let a  Dr, and let b 
have that a = HqeF g(xq) and b = Hqea g(Yq) where maxF = r. maxG = p. each 
xq e Cq and each yq e Cq. If r < llin a, then ab = qeF g(zq) . qea g(Yq) e Dp. 
Ifr k p, thenab = qeFX{-r} g(xq)'g(xr'qG yq)e Dr (where the qeFX{} g(xq) 
term is simply omitted if F = {r}). So assume that rein G 
G " q  r}, and let L = {q e G  g > r}. Then ab = HqeF{} g(xq).g(x. 
,u).  (u) e D. 
Thus D is a semigroup of idempotents and for each p < A, K(Uq p Dq)  Dp. 
Let r  p, let o = qeF g(xq) e Dp, let b = qG g(Yq)  Dr, and let c = 
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H,eH g(zq)  Dp. Then 
abc = H 
qF\{p} 
= II 
qF\{p} 
qeF\{p} 

qG qH 
(.1. (» . H .q " H z) 
qG qH 

g(.rq)- g(XpZp) 

alld 

(c - 

H g(.rq)-g(.rp- H 
qF\{p} qH 
qeF\{p} 

So abc = ac and so each Dp is a rectangular subsemigroup of D. To see that D v 
is a rectangular component of D, suppose that a  Dp and b  Dq, where q < p. 
Then f(bob)  C and f(b)  Cq, and so b,b ¢ b. To sho« that Dp  IiA I if 
lAI > [,Xl > a, we observe that Cp contains a left ideal L and a right ideal R, each 
with 1,4[ elements. If a,b  L, then ab = a and so g(a)g(b) = g(a). Thus g[L] 
is contained in the left ideal Dpg(b) of Dp. Silnilarlv. g[R] is contained in a right 
ideal of Dp. So Dp contains a left ideal and a right idem each with at least 
elements. They cannot have more than lAI elements because for each F  
with p = lnaxF, there are lAI IFI = lAI choices for 1-[qeF g(Xq). So [Dp[ = 141. 
Thus D v , L x R . IAI- [] 

Two ol»vious questions are raised by Lelnlna 3.4 and Theoreln 3.5. First, can 
the flmction 9 constructed there be required to be continuous? Second. tan the 
flmction 9 in Theoreln 3.5 be required to be a honollorphism? We shall answer 
bot h of these questions in t he negative, even w hen the stronger requirements that T 
and (7 be compact and (7 be a topological semigroup are added. We shall ha:ce lmed 
of the following lemma, whose routine proof we omit. (Recall that anv successor 
ordinal is a COlnpaçt Hausdorff space under its order topology.) 

Lemma 3.6. Let .4 be a compact space, let A be a ordinal, let A x A x (A+ 1) bave 
the product topology, avd let A and A x A x (A + 1) be clopen subsets of C = CA.X. 
Thev (: is a compact topological semigroup and (' = K(C). 

We now show that. eVell for A = 0, one cannot require that g be continuous. 
"Ve remind the reader that an F-space is a completely regular space A" in which 
{x  X : f(x) > 0} and {x  X : f(x) < 0} are completely separated for ail 
continuous f : X  

Theoreln 3.7. There eïist a nonempty set A. a topology o'n C = CA, 1 8uch that 
C is a compact topological semigroup and A and Co are compact subsets of C. a 
compact right topological semigwup T. and a continuous surjective homomorphism 
f : T -- C such that there is no cottinuous homomorphism g : 6' --} T for which 
f o gis the identity on C. (In fact. there is no cotinu.ous injective function from 
C roT.) 
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Pro@ Let A = pIN, let C = fin U (fin x fin x {0}) with the topology given in 
Lelnlna 3.6, and let T = H. Then N U (N x N x {0}) is dense in C = A(C). So 
there is a continuous surjective homolnorphism f  T  Ç by Theorem 2.5. 
Now suppose there is a continuous injective flm«tion 9 "   T. Then by 
Theorem 2.2 there is a continuous injective flmction ff'oin Ç to  ç fiN. But this is 
ilnpossible because fin is an F-space [3, Theorem 14.25]. So evel'v compact subset 
X of/IN is an F-space, because everv COlltillUOllS function fronl .ç to [0,1] ll a 
continuous extension to fiN, by the Tietze extension th(,oroln. But fin x fin is llOt 
an F-space by [3. 14Q]. 
Theorenl 3.8. There exisl a nonempt set A u,ith distinquished el«ment 11, a topol- 
o9 on C = CA,+I suc tat C is a compact topological semigrou p aud A 
are compact su.bsets of Ç for each p  ', a compact rtgt topological semigroup 
T, and a continuous surjective homomorpisnt f " T  C suc tot lhere is no 
homomotph.ism 9 " C  T for whi«h f o 9 is the identit.q o ('. 
Pro@ Let A = {0} raid let C = CA.+a with the tolmlogy given in Lemma 3.6. Let 
u0 = 0. forp < w, let up+a = (O.O.p), and let u = (0.0. w). Th(,n C = {p "p N w} 
811d Upq = llpv q fi)l" ail p, q  w. "[blologically, u is the onlv non-isolated point 
in C Let (vp)v<  be a sequence of distinct points none of which are in C. Let 
T = {u v  p < ,} U {vp  p < «} and for p.q < ' define an operation on T as 
follows: 
IIpq 
IIp'q 
l'pq  Up'q  p . 
b leave it to the reader to verifv that the operation is associative. 
Let ç {e0} be discrete and let T be the one point compactification of T {v0} 
(with t'0  the point at infinity).  claire that T is a riht topological semigroup. 
Let p < . To ste that p is contimous at v0. let Il be a lmighborhood of 
'0 = (0) .d t U = U'({" 
To see that p, is contimous at v0. let II be a neighborhood of v0 = P,(0) and 
t u = {% . p  q <  ..d  
Define f" T Ç by f(up) = up and f(vp) =u. for eachp < w. Then f is a 
continuous surjective honlomorphisln. Suppose that g  C ç is a honlomorphisnl 
for which fog is the identity. Then for p < 
such that g(u) = vq. But then. vq+ = 
g(u) = eq. a contradiction. 
 shall see next that we can get the function 9 to be a honomorphism bv 
requiring that T be Selnitopological. This corollary tan then be viewed as saying 
that Ç is something like an absolute co-retract in the category of semitopological 
semigroups. Ç becomes an absolute co-retract in the category of compact semi- 
topological senfigroups if it is given a topology for which it is in this category with 
A and each Çp being compact. 
Corollary 3.9. Let A be a nonempty set wilh distingished elemenl O. let A be an 
ordinal, and let Ç = Ç.4.a. Let T be a senitopological semigroup, and f  T  Cbe 
a continuous homomorphism such that f-'[A] is compact and f-[Cp] is compact 
for every p < A. Then ther« is a homomorphism g ' C T such that f o g is the 
identity. 
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Proof. At stage p of the induction in the proof of Theorem 3.5 one has that for 
each q < p, g(Uv)- f-l[cp] is a compact right ideal of f-[c]. So one may choose a 
minimal right ideal R of f-l[cp] with R Ç q< g(uq), f-[%] and f[R] 
Then. if  G Cq for some q G p and G Cv, just s one showed in the verification of 
hypothesis (i) that 9(x)  9(P)  L- 9(g), one can show that 9(9) "9(w)  9(9t) " R, 
so that 9() " 9(x) and 9(9x) are idempotents in the saine group. If 9  A and 
x  Cp, then 9(U)" 9(x) = 9(u.r) by (ii). ( did hot need to consider the case 
y G A separately at that point in the proof of Theorem 3.5 because the equation 
qr = tlqr '-q valid in any event. The corresponding equation y.reUq = ç(Uq is 
hot valid if y  A.) 
 now prescrit some immediate consequences of Theorem 3.5, although with a 
bit more effort, wc shall get a stronger result, namely Theorem 3.16. 
Corollary 3.10. Let  be an ifinite cardinal and let A be an ordinal with 
Then  cotains a subsenigroup D = UpA Dp of idenpotents wher¢ each Dp 
a rectangular comportent of D isomorphic fo I ad the seqence (Dp)A is de- 
çasing i the ord('ring of çompoent. o that for ea«h p  A. Dp = K(UqS  Dq). 
Proof. Let a have the discrete topology and let 4 = fla-. Let C= 4.+ and let 
lmve the topology describcd in Lemma 3.6. Let T = . Since U( x  x (A+ 1)) 
is a dense sui»set of C = A(C), by Theorem 2.5 there is a continuous surjective 
homomorphism f  T  C and so Theorem 3.5 applies. 
Corollary 3.11. Let ,S' be n infinite cancellative discr¢te semigroup with cardi- 
ality  ad let A be an ordi,al with lAI G a. Then ils  S contabs a subsemigrop 
D = UpA Dp of id¢mpotents where ¢ch D is a r¢ctngdar coponent of D 
isomovphic fo l/ and the seqence (Dp)pA is decreasing i the ord¢ring of com- 
o¢t., o t.t fo ¢« p  . Dç = ç(U; G)- If s = (N, +), s = (N,-), o 
S is  coudabl nfinite discrete gmp. then D ç K(flS). 
Proof. By Theorem 2.7, ils  S contains a topological and Mgebraic copy T of . 
(If S = (N, +), choose T = . If S = (N,-) or S is a countably infinite discrete 
group, choose T  in Theorem 2.4 or Theorem 2.3 respectively.) Then Corollary 
3.10 applies. 
If S = (N,+), S = (N,.), or S is a countably infinite discrete group, then 
K(T) = K(çS) aT So bv Theorem 3.5, with A+ 1 in place of A. we bave 
[c] ç K(ç)  ç(S) ana so D ç K(S). 
Corollary 3.12. Let S be a contabl infinite discute grop. Then there is a copy 
of ç« contained in K(flS). 
K can completely clmracterise the semigroups of idempotents that can be cm- 

bedded in K(flN). 
Corollary 3.13. 
nality t and let D 
(i) There is a 
(ii) There is a 
Proof. Conclusion 
3.11. Assulne now 
[D I _< I/3NI = 2'. 

Let S be an infinite cancellative discrete semigroup with cardi- 
be a semigroup of idempotents. 
copy of D in ris \ 5? if D is rectangular and lOI < 2 2" . 
copp of D in K(flN) if and only if D is vectangular and [D[ < 2'. 
(i) and the sufficiency of (ii) follow immediatelv from Corollary 
that D is a semigroup of idempotents contained in K(flN). Then 
Next observe that any subsemigroup of idempotents in K(flS) 
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must be rectangular. To sec this, sui)pose that a', y, z E K(13ç). Then zz and r9z 
belong to t.he saine minilnal left ideal and t.o t.he saine nlinimal right, ideal. Hence, 
if they are i(telnpotent, they lnUSt, be equal. 
Recall that any two nlaXilnal groups in the slna.llest ideal of a COlnpact right 
topological semigroup are isomorI)hic. çç sec that we can get the direct product of 
such groups with an embcdded rectangular Selnigroup iii tlle slnallest ideal as well. 
Theorem 3.14. Let T be a compact right topological sew, igroq», let D be a rect- 
a9u.lar su.bsemigroup of K(T), ad let G be a vm'inal subgrou p of K(T). There 
is an algebr«ic copy of D x G covtaived "i K(T). 
Proof. Let L be a lnilfimal left i(leal of D and let R be a lninimal right ideal of 
D. Since D is reçtangula.r, D is the internal direct product of L and R, lnealfing 
that each element x of D can be written mliquely as x = xx where x G L 
and xn G R. Also, RL = R  L is a subgroup of D and so, sinçe D consists of 
idelnpotents, RL = {e} for SOlne e. Then for any x,y  D, ,rnyc = e. Note a.lso 
that (xy) = xc and (.ry)n = y. 
e lnax" assume th«tt G = etc. Dofine  : D x G  K(T) by (x, g) = 'Lg,rR. 
 claire that  is an injective honlomorphism. Let (, 9), (Y, h) 6 D x G. Then 
ç((x, g)(y, h)) = xLg.rnyz hyn = .r Lgehyn = (,ry)Lgh(.ry)R = ç(,ry, gh ). 
Now assume that (x, g) = 9(Y, h). Then g = ege = ,rRxrg.rR.rL = .rnyrhyR.rL 
= che = h. Also, xLT  yLT ¢ 0 and xLT and yLT are minimal right ideals of 
T; so xLT = yrT. Similarly T.rn = Tyn. Now .r = 'L,rR G .rrT  Tan and 
  yLT  TyR. So ,ç and y are idempotents in the saine gronp and therefore 
,r= y. 
Corollary 3.15. K(N) cotains an algebraic copy of 1, x F, whe I5, is the 
2 « x 2 « rectangldar .semigroltp ad F is the free group on 2  genemtors. 
Pmof. Bv Corollary 3.13, K(N) contains a copy of the 2  x 2  rectangular Sllli- 
group, and by [4], each maxinlal group in K(N) contains a copy of the free group 
on 2  generators. Therefore the result %llows Kom Theorem 3.14. 
We now present, a strengthening of Corollarv 3.10, producing a longer chain of 
rectangular components. ecall tha/ the Sosliz zumber S(X) of a topological 
space X (also known as the cellalarity of X) is the least cardinal 2 such that. X 
do no/ bave a collection of 7 pairwise disjoint nonempty open subsets. Sec [2, 
Chapter 12] for considerable information about the Souslin number of t.he space 
U(n) of uniform ultrafilters on -. ecall. in particular, t.hat the Souslin number of 
N* = çNN = U(N) is 
Theorem 3.16. Let  be an infinite cardinal and let  be an infinite ordinal for 
c I[ < S(U()). There ezist a set A with lAI = 2 " ,,d , to g 
if p = q+n for some  < w, then 9(g)'9(x) = 9(gx). Also,  contains a semigroup 
D = p< Op of idempotents whe for each p < , Op is a rectagular comportent 
of D isomorphic to ç, 9[Cp] ç Dp, and the sequence {Dp)p< is decasin 9 in 
s.ccessor, then D_ ç K(). 
Proof. Since lAI < S(U(n)), choose a fanfily {Ep)p< of subsets of  such that each 
[Ep[ = n and Ep  EqI < ¢ when p ¢ q. For each p < A we define 
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by bp(w) = Yeoepns,pp(,) e (where e0 e = 0) and let 
be the continuous extension of çp. If v, u,  11 and supp(v)  supp(w= 0. then 
0p(v + u,) = çp(v) + 0p(w). So by [6. Theoreln 4.21] the restriction of p to N is 
a hollloHorphisllL 
Next observe that for x  N, p(z)  {0}UN. Ifthere exist B  z and 
o < n such that supp(u,) Ep = 0 whenever w ¢ B and rein supp(u,) 
then ç(x) = 0 because çp is constantly 0 on {u' ¢ B  rein supp(u,) 
Otherwise {0p[{« e B  rein supt)(w) OE }1   e - .d  < } has the finite 
intersection property and so is contained in ail ultrafilter . This   N and 
Let T0 = N  c?{w  II  supI)(U,) ç E0}. Notice that T0 is a compact 
subsemigroup of N. For each p with 0 < p < A let 
T o= {x Ç N  çp(x) e N and for all q with p < q  A, çq(w) = 0}. 
To sec that T v ¢ , let .r be a mfiform ultrafilter on {e  a  Er. If q ¢ p, then 
[Eq  Er[ < . So çq(.c) = 0. while çp(x) = z  N (because çp is the identity on 
{«  a:  Ep}). Since q is a homomorl)hisln on N for each q  A we has'e that 
Tp i8 a s/l])Sellligrollp of . Sillce Tp :   -1[]  p<qA 
compact. 
If A is a successor, let T_ = N  v v-[N]. Then T_ is clearly 
compact subsemigroup of N provided T_ ¢ 0. XX show in fact that 
T_. Let x  K(N), let p < A. and let 9 be a uniforln ultrafilter on {e  
a G Ep}. By [6, Theorem 4.39] pick z G N such that m = z+9+w. Then 
0,«) = ç,:) + ç,u) + ,) = ,) + v + ,) ¢ 0. 
Next observe that for p, q < A, T v+ Tq = Tpvq and if p ¢ q, then T 
Let T = p< Tp. If T has the relative topology induced bv N, T is a right 
topological senligroup. 
Let A = N  cg {e "   E0}. Then A is exactly the set of uniform ultrafilters 
on {e "a  E0}, and so IA : 2 2. Let C = Ca.. 
X% shall now construct a smjective homomorphisn f  T + C XX first intro- 
duce some mappings. Let 0  II  {e  a  0} be a flmction whose restriction 
to {e  a  E0} is the identity, whose restriction to {e  a Ç E} is a bijection. 
and whose restriction to II  {e  a  E0 U E} is a bijection. (In particular. 0 is 
at most three-to-one.) Let 0  1I  d {e  a  E0} be the continuous extension 
of 0. 
Let e(0) = a(0) = 0. For w e I1{0}, let e(w) = e where 7 = max supp(w). If 
supp(w) ç E0, let a(u') = 0. Otherwise let a(,) = e where o = min(supp(w)E0). 
t -  + {0}  d{« - < «} ,,a - d  {0} u d 
the continuous ext.ensions of a and e respectively. Notice that a is the identity on 
{e " O e E0} alld e is the identity on {e " a < }. So ? is the identitv on 
N  d {e  a Ç hE0} and 7 is the identity on N  cg {e  a < -}. X claire 
that for «.  G 

-¢(x + y) = è(y), 

(,) 

(.r+B) =  (x) ifxTo, 
(y} if.rTo. 
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For ¢v.v E II,, if max supl,(V) > max supp(w), then ((w + v) = e(v) o that 
è(w+V) : ê(U): if supp(w) ç E0, then 6(u,+v) : 6(v) so that (w+g) : (U). For 
w, v G II', if max supp(w) < lllill supp(v) and supp(w)E0 # 0, then a(tv + t,) = 
d(w); so à(w+ ) = d(w). The equations in (*) then follow bv the continuity of 
Vr .  , . .a." = a(g(.,-}). , < p <  and .r G 2;, let 
because & is at most three-to-one.) Thus fiT0] 
and f[Ts] ç Çs if w 5 P < A. 
({ - 
p < A and let (.e,p)  Çp. Ifp < w, let q = p+ 1; otherwise let q = p. Pick 
g'  c( {e  e  E } such that #(j) = . Pi«k .r  Tv. Then ,' + a" + :  Tq and 
/{' +. + ) = ({({{' +  + )).(«{' +. + )),p) 
= 
= (('), (:).) 
= (, ,). 
Whl-«O-,/[d = C. 
The verification that is a holnomorphisln is routine using the equations (,). 
Çhoose   Ç  T 811d (Dp)<  gua.ranteed by Theorem .5. Since we 
have alreadv observed tha lAI = 2- . a.ll conclusions follow ilnlnediatelv except 
the assertion tha D_ ç K() when A is a successor. To see this recall that 
K() Ç T_ 1 8o tl18 K) 
Theorem 8.5(iv), [Ç-I] 
thus an idem of D and therefore D_ = K(D) 

Corollary 3.17. Let A be an ordinal for which ]AI = c. There exist a set A with 
lAI -- 2 c and an injection g : C.4.) --, IHI such that if q < p < A, y E C e, ad .r Ç C v, 
th« g(a').g(y) = g(a'y), a,,d if p = q+ n for som« n < , tb«n g(y).g(,) = g(y.r). 
Also,  contains a semigroup D = p< D v of idempotetts where for each p < A. 
Dv is a rectangular «onwonent of D. g[Cp] çDv. a,,d the sequen«e (Dp)v<x is 
decreasing in the ordermg of components. For each p < A. [DpI = 2 « and f A is a 
successor, then D_ ç K() ç 

Pro@ Bv Theorem 2.2, NI and NI are topologically and algebraically isomori)hic. 
Also S(U(w)) = c +. So this is an ilnmediate conseqllence of Theoreln 3.16. [] 

It was shown in [5. Corollary 3.4] that there is a _<L-chain (Uo-)o-(tOl of distinct 
idempotents in ON with the property that for each o- < w, U«+l _< 'u«. \Ve are 
uow able to establish a considerably stronger statement. (The necessity in the 
following corolla.ry was also established in [5], but xve includc the short proof for 
completeness.) 

Corollary 3.18. Let  be an ordial. There is a <i-chain (u«)«< of distinct 
idempotents in fin with the lnvperty that for each a < A. l/a+1 __ tic if attd onl'!l 
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,if lAI _< «. /f lAI _< « ard  is a ,,.«««so. o,.« ,,, cho.»¢ «h a .,«q««¢ ith 
Proof. Necessitv. For each « < A, N* + .« properly contains the compact set 
N* + «+. So one can choose a clopen subsct U« of fin with N* + U«+l ç U« and 
(N* + u«)U«  . The clope subsets of 3N correspond extly to the subsets of 
N ad so there are exactly « of them. 
Suciency. Choose A and 9 as guaranteed bv Corollary 3.17 for A. For each 
p < A, let u v = 9(0.0.p). If A is a successor, then ua-1  9[Ç-] ç DA-I ç 
ç().  
Question 3.19. Is therc a dccreasing 5-chain of idempotents in N* indexed by 
w+ 17 
X close this section by observing thnt it is consistent with ZFÇ that there are 
kempotents in N that are hot members of any notrivial rectangular subsemigroup 
of N. Iadeed, by [6, Theorems 12.19 and 12.29 and Lemma 12.44], Marth's 
Axiom hnplies that there is nn idempotent p  fin such that. whenever q  N, 
r  ,=l«B(t), and p= q+r, one must havep = q = r. In particular. f 
p = p+ q +p, then p = q. 
I can be shown h ZFC that there are idempotents p in  that are strongly 
right maxhnM; .e., the e(luato q + p = p, vth q  N, imples that q = p 
[6. Theorem 9.10]. If p s an dempotet of this kind. p does hot belong to any 
semigrop h  isomorphic o a semgroup of the form 1/ mdess [AI = 1. 
. CHAINS OF RECTANGULAR SEMIGROUPS AS CO-RETRACTS 
It was shown in [10] that certain infinite chains of finitc rectangular senfigroups 
are absolute co-retracts. X prove in this section a shnflar theorem in which the 
the rectangular semigroups are allowed to be infinite. As a consequence, we obtai 
additional semigroups whk.h can bc algebraically embedded in . 
Definition 4.1. Let A = (A}< and B = (B.}< be sequences of sers. Assume 
that each A, bas a designated elcment a and each B bas a designated element 5. 
Suppose also that, for each, < w, either A, = {a} or B = {5}. For each p <  
we define Dp to be the set of pairs of words of the form (a0al -.- av, bvbv-1  .. bo), 
where ai  Ai and bi  Bi for each i  {0.1 ..... p}. For 0 < A  w, we let 
D.n. = < Dp. We define a semigroup operation on DA.n. as follows: if 
.r = (aoa...ap, bpbv_...bo)  Dp and g = (coc...c,dd_l...do)  D, where 
q  p, then 
xy = (0al-.-p, bp---b+ldd_l-.-d0) 

and 

g3 -- (CoC1 "   cqaq+l.., ap, bpbp_l .-. bo). 
We leave the verification that the operation is associative to the reader. Observe 
that each Dp is a rectangular semigroup. 
Notice that if A is a set with designated element 0. A0 = {0}. B0 = A. A1 = .4, 
B1 = {0}, and A, = B, = {0} for n > 1, then D.4.., is isomorphic to CA.,» (Send 
(0. a) to a and for p > 0 send theelement (0a00---0,00-.-0b) ofDp to (a, b. p -1).) 
Thus the structure of D.ts., is, in general, considerably more complicated than 
that of 
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Definition 4.2. Let p < w and let x = (o0o 1 "''(lp, bpbp_l ... bo) E Dp. We define 
elelllellt.s 1() and 2(,T) in Dp by çl(X) = (01 "..p-l(tp,pp-1 ""0) and 
2(X) = (GOal ""ap, bp5p-15p-2"'" 50) and if p > O, we define .re and x in Dp-1 
bv xe = (oOal...Op_l, p-lp-2"" o) and x = (o1 ""p-1, bp-lbp-2"'" bo). 
 put u v = (o"" 6p, 5V5V--l''" 0)  D v. 
 show that D is somet.hing like an absolute cretract.. 
Theorem 4.3. Let A = (A)n< ad B = (B)n< be sequences of sets o.s i 
Defiitio .1, let 0 < A  w, artd let D = DA... Let T be a rigt topological 
semigroup, a«td let f " T  D be a sue«tive owmorphism such thot f-l[Dp] is 
copact for eac p < A. Ten tere is a homomorpism g  D  T for wich f  g 
is te ide, tity. ff T is compact atd A < w, ten g[D-l] Ç K(T). 
Proof.  mav assume that A0 = {a0} so that D0 is a right zero semigroup. 
Exactly as in the first paragraph of the proof of Lemma 3.4 we can define g  D0 + T 
such that g is a homomorphism and f  g is the i<lentity on D0. So we assume that 
p > 0 and g has been defincd on q<p Dq. 
For each «  Dp, note that «Dp is a minima.1 right i<leal of Dp ami Dpxis a 
minimal left idem of Dp. So we mav «hoose a minimal right ideal R(«) of f-l[Dp] 
and a minimal left idem L(x) of f-l[Dp] such that f[R(«)] = XDp and fiL(x)] = 
Dp.r. Givcn x e Dp, we have by Lemma 3.3 tha[ g(xe)R(¢(x)) is a minimal right 
ideal of f-[Dp] and L(¢2(x))g(x) is a minimal left ideal of f-l[Dp]. So we may 
define g(x) to be the identitv of the group g(xe)R(¢(x))L(¢2('))g(xr). Notice 
that if T bas a smallest idcal (in particular if T is compact) and A = p + 1, then 
h-(W) ç f-lirai. S+ ç(f-[V]) ç ç(W) .,,d tm g[V]  ç(W). 
+w f(a(«)) e ,.¢(«); = « a,,d f(a(«)) e ;¢(«)' = «. S+ 
is an idempotent in 'DpX and thus f(g(x)) = x. 
Suppose that x e Dp and y e Dq where q ç p. Then ¢(x) = ¢1('Y) and 
ce = (')«. s+ a(«) e (««)(¢1()) ,d a(«)a() e ('e)(¢l(«))(p) ç 
g(xe)R(¢(x)) by Lemma 3.3. Thereforc, g(xy) and g(x)g(y) are members of the 
saine nfinimal right ideal of f- [Dp]. 
If  < , ¢() = ¢() d () = «. s+ («) e z(¢(«))a((:)) 
a(«)a(u) e L(¢(:))(:)a(u)= (¢:(:))a(:)= (¢(-))a((-)). 
If  = p, ¢() = ¢(y) a,,d («) = y.. S («) e (¢:(y))a() 
me minimal left ideal of f-[Dp]. 
By a left-right switch of the al»ove arguments we have that g(yx) and g(y)g(x) 
are in the saine minimal left idem and the saine minimal right idem of f- [Dp]. 
First assume that q < p. Pick a e R(¢(x))L(¢2(x)) such that g(x) = 
g(xe)ag(xç). Thon 

g(x)g(y)g(x) 

= g(xe)ag(x.r)g(y)g(xe)ag(x,) 
= g(xe)ag(x,y.re)og(x,) 
= («)g(xxe)(xT) 
= g(xe)ag(x)g(xe)ag(x,) = g(x)g(x) = g(x). 

So g(x)g(y)g(x)g(y) = g(x)g(y) and g(y)g(x)g(y)g(x) = g(y)g(x) and thus g(x)g(y) 
= g(xy) and g(y)g(x) = g(yx). 
Now assume that q = p. Assume also that Bp = {5p}. (The case that Ap = 
then proceeds by a left-right switch of the following argument.) Then ¢2(x) = 
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_( = . A.o . = .« = ._. Th,,, 9(.)9()  L(:(.r))9(x)g(y«) = 
L(up)g(tp_) and g(y)g(xt)  L(up)g(tp_), which is a lninimal lcft ideal of 
f-l[Dp] by Lelllnla 3.3. ' have alleady verified (ha(g(.r)g(ye) alld g(y)g(xe) 
are idempotents. So, since they are idempoten/s in /he saine minimal lefl ideal 
(.)(y«)()(«e) = ()(e). we bave 
(:.)(.q)g(.) = g(..)(,..)(xt.r) 
= (.)g(«)g()(..)g(.) 
= (.)(.'»'t)(.) 
= g(..,»)g(.«) = g()g(.) = g(). 
(',onsequently, g(.r)g(y) aml g(y}g(x) are idempotonts. 
Corollary 4.4. Suppose lhal nis a infi,ite cardiml and lhat eoch 
«ith«r {0} or 2 2. Thon DA.. tan be embedded in. . 
Proof. N»r each p < , we give Dp t]le topology defined by regarding Dp as a 
subspace of (fin 2p+2. We define the topology of D bv ta.king each Dp to be clopen 
in D. Then D is a topohgical semigroup vith a dense subspace of cardinality 
The conclusion then follows from Theorems d.3 and 2.5. 
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ABSTRACT. It is shown that, for a rninimal and integrable action of a locally 
compact quantum group on a factor, the group of autornorphisrns of the factor 
leaving the fixed-point algebra pointwise invariant is identified with the intrin- 
sic group of the dual quantum group. It is proven also that. for such an action, 
the regularity of the tixed-point algebra is equivalent to the cocommutativity 
of the cluantun group. 

l. INTRODUCTION 

When given an action o of a locallv COnlpact quantum group G on a von Neu- 
mann algebra A, one may associate to it the subgroup Aut (A/A ) of all autonlor- 
phisms of A leaving the fixed-point algebra 4 ' invariant pointwise. Let us call 
this subgroup "the Galois group of a". As [1, Theorem III.3.3] suggests, it would 
sometimes happen (or be expected) that the Galois group carries an important 
piece of information on the quantmn group G itself. With this philosophy in mind. 
we started in [17] to investigate Galois groups of lninimal actions of coinpact Kac 
algebras on factors by making good use of the Galois correspondence established 
by Iztuni, Longo and Popa [7]. In [20], we succeeded in describing the Galois grottp 
of any minimal action of a compact Kac algebra as the so-called intrinsic group 
of the dual discrete Kac algebra. This extended the result of [1] cited above. As 
an application, we were able to show that. if the quantum group in question is 
finite-dimensional, then its cocomlnutativity is equivalent to the regularity of the 
fixed-point algebra. Our main goal of this paper is to extend these results to a 
larger class of locallv compact quantum groups, hOt only compact Kac algebras. If 
we trv to achieve this goal exactlv along the line carried out in [17] and [20], then 
a Galois correspondence for a (lninimal) action of a more general locally coinpact 
quantum group would certainlv be needed. At the moment, it seems that the re- 
sults of Enock in [4] would answer this purpose. Unfortunately, there are however 
a few lnistakes in his proofs, and. to the best of the author's knowledge, they have 
hot been restored yet. So we cannot apply Ellock's Galois correspondence to the 
situation we will consider in this paper. Therefore, we will adopt a new approach 
here that does llot resort to anv Galois correspondence. 
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The outline of this paper is the following. In Section 1, we fix the notation used 
in the whole of our discussion. Basic fa«ts about, locally COlnpaet qllalltUlU groups 
(iii the sense of Kusterlnans and Va.es) aud their actions on von Neumann algebras 
are collected. In Sect.ion 2, we will prove that the Galois group of a milfimal, 
int.egrable action of a locallv compact quantum group G is topologically isomorphic 
to the intrinsic groul) of the dual G. Section 3 is concerned with regularity of the 
fixed-point algebl'a of a lninimal, integrable action. We prove that the regularity 
considerably deterlnines the structure of the qUalltUlll grollp. Naluely it is shown, 
with some exception, that the fixed-point algebra is regular if and only if the locally 
colnpact quantum group under «onsideration is cocomnmtative. In Section 4. we 
lnake a few renmrks on the Izmni-Longo-Popa Galois correspondence. Olm of them 
concerns an explicit formula fol" the inverse lnap of their Galois correspondence. 
Finally, we includc an Appeudix fol" SOlUe auxiliary results which are applied to the 
argument lnade iii Section 3. 
The author is grateful to Professors Michel Enock and Stefaan Vaes for having 
ilfformed hiln that there are mistakes in SOllle proofs iii [4]. He is also indebt.ed to 
Professor Masaki lZUlni for indicating an crror in the earlier draft, of the manuscript. 

2. TERMINOLOGY AND NOTATION 

(àiven a VOll NetlUlallll algebra A and a faithfld llOrlllal semifinite weight ¢ Oll 
A, we introduce the subsets n, rn and rn of A by 

* + = me N A+. 
n = {x  A" ¢(x*x) < oe }, me = rionS, rn, 

The standard (GNS) Hilbert space obt.ailmd frOlll ¢ iS denoted by H. We use 
the sylnbol AO for the canolliCal embedding of ne# into H 0. The modular objects 
such as the lnodular operator, the lnodular conjugation, the S-operator, the F- 
operator, t.he modular automorphism group, etc. associated t.o çb are denoted bv 
Ve, 34,, S¢, F¢, a ¢, respectively. (Since we follow the notation enlployed in [10], 
the sylnbol Vwill be used t.o deuote the lnodular operator of a weight.) The set of 
unitaries iii A is denoted by H(A). For a von Neulnallll subalgebra B of A. define 
JV'(B) := {u  H(A) : uBu* = B} and call it. the normalizer (group) of B in A. 
V:e let B(H) stand for the algebra of ail bounded operators on a Hilbert space 
H. 

2.1. Locally compact quantuln groups. 
lï)efinition 2.1. Following [10] (see [9] also), we sa,v t.hat, a quadruple G = (]il, 
A, 0, b) is a locally compact quantum group (in the von Neulnann algebra 
sett.ing) or a von Neumann algebraic quantum group if 
(1) /11 is a von Neumann algebra; 
(2) A is a unital normal injective .-homomorphism from M into M ® M sat- 
isfying (A ® id) o A = (id ® A) o A: 
(3) o is a faithful nornml selnifildte weight on kl such that 
ri1+ - 
o((w®zd)(A(:r)))=o(z)w(1) (VwM. +, V:e ), 
(4) /, is a faithful normal semifinite weight Oll Al such that. 
/,((id ' w)(A(:r))) = #,(a:)(1) (Vw  M. +, Vin  më). 
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Let 118 fixa localh, compact qtlalltllln grotlp Ça; : (]ll, A. ç, b) throughout the 
rest of this section. We will always think of M as represented on the GNS-Hilbert 
space H obtailied ffoto . By the left invariance of 
on H  H characterized by 
II'(G)*(A(x) @ A(y)) = A$(A(g)(,r @ 1)) (x,g  n). 
This unitary is called the Kac-Take.soki operator of G, and is denoted silnply by 
Il if there is no danger of confllsion. The modular operator and the lnodular 
conjugation of  will be denoted simply by Ç a.nd J. The unitarg atipode, the 
scalig gronp and the .scalig co.tmd of  are respectively denoted bv R, {7t}tER, 
p() 0). As iii [10], wç aS81llll that @=ço R. 
According to [10]. there canolficallv exists another locally compact quantmn 
group â = (kî, , , ), called the locally compact quaztum group dual to a such 
that {M. H} is a standard lepl'esentation. So we always regard 
In fact. I is by definition the von Nenmanl algebra gcncrated by {(w @ id)(ll )  
w  M,}. The lnapping : w  M,  (w @ id)(lI')  I is called the Icft regular 
repre.settation of G. There is a canolfical identification (= the Fourier trllsforlll) of 
Hç with H. So we conskler thc lnodula.r operator and the lllOdtllar conjugation of 
, denoted by Ç and ,Ï, as acting on H. The mfita.ry a.ltipode, the scaling group 
and the lnodular elelnelg of  are denoted respectively by , , .  say that 
G is COlnpact if (1) < . In this case, we agree to take  to be a state. X say 
that G is discrete if G is conlpact. For the defillitions of locally compact qllalltUln 
groups such as the COllllllllta.llt t, the opposite G °p etc., we refer the readers to 
[10, SectiOll 4]. 
It is known that every locally colnpact group F canollically gives rise fo a COlll- 
nmtative locally conlpact qtlalltlllll group whose tlnderlyillg VOll Netunalln algebra 
is L(F). XVe denote it by G(F). The underlying von Neulnalm algebra of the 
dual G(F) is the group von Neunmnn algebra of F gelmrat.ed bv the lefl regular 
repl'esentation of F. 
 denote by /G(G) the set of all unitaries u  I satisfying &(u) = u @ u. 
The group IG(G) is called the intrinsic group of G. Next defille Ç(G) to be the 
group ofall automorphisms fl of I such that (fl@id) o& = off. By [2] (see 
[19] also), /G(G) is topologically isolnorphi« to Ç(G) through the mapping: v 
IG()  fl := Ad v[  Ç(G). Here IG() is endowed with the strong-operator 
topology, and, for a general (separable) von Neumalm algebra P, we consider on 
the automorphisln group Aut (P) of P the topology of silnple COllVergellce Oll the 
predual. It is kllOWll that v is the canonical implelnentation of fit,. 
ç soEv that N Ç I is a right co-ideal (von Nemnalm subalgebra) of G if 
is a von Neumann subalgebra of I satisfying &(N) ç N @ ,I. Thanks to [4, 
Théorèlne 3.3], we know that N Ç I is a right co-ideal of G if and only if Olle has 
(2.1) N = Ma(MaN')'. 
A (le) action of G on a von Neunmlm algebra A is a normal injective unital 
 -homolnorphism a from A into kI @ A satis'ing (id 
([11). 
Fix an action a of G on a von Netllllllll algebra A. By [16. Proposition 1.3], the 
equation 
%(.):=($ia)((a)) (.+) 
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defines a faithflfi normal operator-vahmd weight T, fl'OIll 4 onto .4  := {a E 4 - 
c(a) = 1 ® a}, the fixed-point algebra A  of c. \Ve call T the operator-valued 
weight associated to the action o. 
The ClOssed produ«t of A bv the action  is by definition the von Neulnann 
algebra generated bv c@4) and ]Î  c. We denote it. bv G « < A. By [16. Propo- 
sitioli 2.2]. there exists a unique action a of op on G a < A. called the dual action 
of a, such that 
(G«< .4) a = n(A), &(z ( 1) = A°V(z) ® 1 (z e M). 

For everv faithful normal semifinite weight b on A. by using the operator-valued 
weight T associated to the duai action d, lhe equation 

By [16, Theoreln 2.6], therc is a unital ,-isomorphisln O from the double crossed 
l)roduct (°Pa < (Ge < A) onto B(H)® A. and an action d of G on B(H)® 4 
such that 

g := Ad (El'*2 ® 1) o (er  id4) o (id(B,,) ® a). 
(Ad (Jj) ® 6) ) o  = d o 6). 
where 1" := (J ® J)EII'*z(.Ï  .Ï) and : H, ® H --, H v 0¢ H is the flip. \' call 
à the stabilizatio of a. 
We say that the a.çtion a is integrable if T is semifinite. The action a, is said to 
be minimal if AV/(A) ' = C and the linear span of {(id"3ca)(a(a)) : o  A. ,,_, G A,} 
is o--weakly dense iii M. 
Finally, G is called a Kac algebra if ri = id a.nd er4' = aV. For the general theory 
of Içac algel)ras, refer to [6]. 

3. REALIZATION OF INTRINSIC GROUPS IN AUT (A/A ) 
Given a von Nelllllallll algebra P alld a von 1Neulnallll subalgebra Q of P, we 
define Aut (P/Q) t.o be the group of ail automorphisms of P leaving Q invariant 
pointwise. 
Let  = (M, A, ç, ) be a locally compact quantuln group. Suppose now that 
we have an action a of G on a von Neulnanl algebra ,4. Throughout this paper. A 
is alwavs assumed t.o be a non-type In factor (n  N). 
The lnapping considered in the following proposition is essentially observed bv 
Enock and Schwarlz in [5] as a special case of their constructions of certain lnor- 
phislns associated to an action of a Kac algebra. The lnapping is still defined even 
for a general locallv compact quantum group. 
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Proposition ;3.1. There e.rists a "uniqtte co,ttinuous hontomorphism front Ç(G) i'rtto 
Aut (A/A ) su«h that, with O the image of fl  ç(G) under this homomorphism, 
(J  d) o., =  o 
If the action o enoys the propert that the linear span of {(idM  w)((a)) : a G 
A, w G .4,} is ¢-'u,eaklp dense i .I. the the above homomorphism is ijective. 
Poof. net fl ç(G) a,,d ,, e .-. Se X := (d  i.«)  () e t  4. Then 
(idM  )(X) = (idM  ) o (fl  idA) o (o) 

= (d @ id^t 
= (13 @ id^t 
= (A®ida 

® id4) o (id^l ® ) o o(a) 
® idA) o (A ® idA)  c(a) 
) o (fl ® ia «) o (,,) = (A  id 

From [16. Theorem 2.7], there exists a ,inique element 03(o ) 
(3.1) (fie id4) o (a) = X = (0fl(o)). 
It is easv to (']w«k bv using (3.1) Hmt 03 is a hOmOlnorphism rioto A into itself. 
Moreover, one tan easilv verify that 0d,& = 0d, o 0& and Oid = id. Hen«e 0 is 
ail automol'phism of .4. That the restriction of 0 d to .4" is the identitv fo]]ows 
also from (3.1). Thus the lnapping fl G Ç(G)  00 G Aut (A/A") is indeed a 
homomorphism. Be«ause of (3.1), we find that (fl id4)[(a) is an autolnOllflfiSln. 
With this in lnind. 0, bas the form 
0. = - o (fl 
Henee fl  0d is continuous. 
Now suppose that the linea.r si)an of {(idt OE )(o(a)) :  G A.  G A.} is 
«-weakly dense in M. If 0 = id. then (3.1) implies that fl is the identitv on 
{(idt  )((a)) :a G A.  G A.}. So  = id. Consequently. the homomorphism 
in question is injeetive. 
Lenmm 3.2. Let fl be an automorphism of 3I such that there is a 0 G Aut (A) 
such that (fl @id) =  0. Suppose that the lmear san of {(idat  ¢)(((a)) : 
a G A.  G A.} is -weakly dense in M. The fl belongs to Ç(G). ad one bas 
0 = 
Proof. Note first that, if 0 is an automorphisln s al)ove, then it automatically 
belongs to Aut (A/A'). Therefore, it suffices l)v Proposition 3.1 to show tlmt fl 
belongs to Ç(G). 
Let a G A and w E A.. Then we have 

03 0 id^ ) o /-X( ( id^l 

/3 satisfies (/3 ® id) o A = A o 3 
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Let  be the trivial action of G on C. Namely, t is the mapping frOlll Cinto 
bI®C defined by,(c) := l@c (c ff C). Then the crossed product G, C is 
(canonically isomorphic to) Î, and the duM action i is the coproduct o. It is 
also clem- that the dual weight of tc corresponds to . In this ce. the stabilization 
ï of t h the form ï(x) = EV*E(1 @ x)EVE for 811y X  B(Hw). Therefore, we 
obt, ain 
(3.2) 6(x  1) =/(x) , 1 
for any x  B( H¢). 
Lemlna3.3. Let z be in B(H¢) such that z@ l G« A. Then z belon9s to M. 
Pro@ Let z be an element as above. Since (B(H¢) @ A) 6 = G«  A. we have 
6(z@l) = l@z@l. Onl,eotherhal,d. by (3.2), wehave6(z@l) =[(z)@l. 
Hence we obtain [(z) = 1 @ z. Since B(H)  = M, z must belong to .I.  
Lemlna 3.4. Suppose that  is minimal. Then we bave (A)'  B(H) @ A = 
M' @ C. 
Pvqf. It is clcm" tha M' @ C is contained in a(A)'  B(Hç) @ A. Take anv 
T  a(A)' B(H)@ A. Since T particularly commutes with any element of 
a,(A ) = C@ A a, it follows from the minimality of a that T belongs to B(Hç) C. 
So it has the form T = y @ 1 for some y  B(Hç). If o ff A and w ff A., then we 
bave 
y(id@w)(a(a)) = (idvw)(Ta(a))= (id@w)(a(a)T)= (id@w)(a(a))y. 
By minimality of a, y nmst be in M'.  
Since (Ga  A)  = a(A), it follows from Proposition 3.1 that there exists a 
contimous homomorphism fl e Ç(o)  O  Aut (Ge  A/a(A)) satisb-ing 
( @ id) o  = & o 0. Since & enjoys the property mentioned in Proposition 3.1. 
the homomorphism fl  0d is injective in this case. 

Lemma 3.5. If the action 
O  Aut (G«< A/a(A)) is 

Pro@ Let 0 be in Aut (G « cE 
O(z® 1)(1®b)= 

is minimal, then the homomorphism fl ff Ç(G °p)  
topological isomorphism. 
A/c(A)). If z E M and b  A . then 
O(z® 1)a(b)=O((z® 1)a(b))=O(z®b) 
O(c(b)(z® 1)) = (1 ®b)O(z® 1). 

This shows that O(M ® C) __Ç_ (C ® A«) ' = B(H¢) ® (A«) '. From this, we obtain 
O(M 6; C) ç B(Hc) ® {A C (A«) '} = B(Hc) ® C. 
Hence, for any z  M, there is a torique flo(z)  B(H) such that 
(3.3) O( z ® 1) = flo( z) ® 1. 
Thanks to Lenlma 3.3, flo(z) belongs to BI. Due to (3.3), it is easy to see tllat ff0 
is ail automorphism of M, and that flolo2 = flOl o flo2, flia = id. 
If a ff A, then 
(3.4) (flo®id)(&(a(a)))=(Bo®id)(l®a(a))= 1 ®a(o)=d(O(a(a))). 
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Fix a faithful nornm.1 semifinite weight w on A, and regard A as represented on 
H. Let .] be the modulm conjugation of the dual weight & and U the canonical 
inlplemeltatiol of o on H @ Hw. So U = ,](,1 @ ,Jw). ('hoose the canonical 
inlplenmntatiol unitary I 
COlmnUtes with ,L, we also have 
It follows front Lemma 3.4 that there exists a unitary v  I' such that I = v @ 1. 
Therefore we have flo(Z) = vzv* for any z Ç ;[. By [12, Proposition 1.9], which is 
valid also for any locally COlnpact quantum gl'OUp, we sec that rt' belollgS to IG(') 
for some r  C with ]r = 1. Since Ioth I" and .ri" are the canonical ilnplementation 
of 0, we lnust have r= 1. So , is in IG(g'). It follows tiret 0 belongs to çP). 
Let z  M. Then we have 
a(0(z @ 1)) = (l)o(z) @ 1) = °P(o(z)) @ 1 
= (/o  id) ()  l = (o .id) d(» ). 
From this, together with (3.4), we get (ff0 @ id) o d = d o 0. By Lennna 3.2. we 
find that 0 = 0 0. Thns we have showll the SUljeclivily. The inverse lnap is also 
«ontinuous due to (3.3). 
Theoreln 3.6. If  is a minimal, integrable action of a locally compact quantum 
group  = (BI. &. ç, ¢) on a factor A. then there exists a topologicai isomorphsm 
Ç e Ç(g)  0 e Aut (A/A ) with the property ( @ id) o  = a o 0. 
Pro@ If .4  is infinite, then, by [16, Proposition 6.4], a is a dual action. Hence 
the assertion follows frOlll Lelnlna 3.5. 
To deal with a general case, take a (separable) infinite factor L, and put , := 
L @ A. Also definc d := (« @ id) o (idL @ a), which is an action of  on ,4 with 
A a = L@A  infinite. Remark that aut (/2 a) = {id eO. 0 e aut (A/A")}. Let 
0  Aut (A/A). By the previous paragraph and the above renmrk, there exists 
13 e çg) such that (floEid)o& = 6o(idLO). But this yields 

4. I={.EGULARITY OF A  IN .4 
AS ill the previous section, let cî be a minimal action of a locally colnpact quan- 
tum group Q = (AI. . ç, ) on a factor A. 
 represent A on a (separable) Hilbert spe K so that {A. K. OE4 } is a standard 
representation. Let u G (A). Then the restl'iction of Ad u to (A)  clearly defines 
an autolnorphism 0u in Aut ((A)'/A'). The holllOlllorphislll u Ç (A )  Ou Ç 
Aut ((A)'/A ') obviouslv has H(A ) as its kernel. The bic extension for 4  ç A 
is denoted by Al. So we lmve A = OE4(A)J4. If A  is infinite, then so is A. 
Thanks to [3, Corollaire 1], we may then choose the above Hilbert space K in such 
a way that there is a unit vector 0 G K that is cyclic and separating for both A 
and A . Let OE4 then denote the lnodular conjugation of A associated to 0- 
Lemma 4.1. Suppose as above that A  is infinite. Then the homomorphism 
u e (A )  0  Aut ((A)'/A ') defined above is surjective, that is. it bas 
Aut((A)'/A ') as its image. 
Pro@ Let 0 be in Aut ((A)'/A'). Since {(A) ', K} is a standard representation, 
there exists a unitary v on K such that 0 = Adv(A),. Since OA, = id, v belongs 
to (A')' = A. It is eass" to see that c is in (A).  clearly have 0 = 0.  
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Remark. ff A is finite and G is finite-dilnensional (so that the .bines index fA " A 
is filfite), then it follows ff'oto [11, Proposition 1.7] that the lllap   (A 
0 e Aut ((A)'/A ') is smjective. 
From now on, we asstnne that a is miuimal and itttegrable (A is hot necessarily 
infinite). Fix a faithful normal semifinite weight « on A. and represent A on 
H now. Let U be the canonical ilntflementation of a associated to «. Due to 
[16, Proposition 6.2], this assmnl»tion is equivalent to the one that ( is outer and 
imegrable. From [16, Corollaly 5.6] and [16, Ploposition 6.2], it follows that the 
inchtsions 
C @ A  ç a(A) ç G.4 and ,4  ç A ç A 
are isomorphic. According to [16. Corollm'y 5.6], the isomorphism p" G a A 
is characterized by 
(4.1) p(a(o)) = o (a e A), 
(4.2) p((ç@id)(ll)@l)=(ç@id)(U*) (ç3I.). 
lncideltally. Equatiou (4.2) can be rewritten as 
(4.3 p('(ç)  1) = (3Jd3  id)(U) (ç  (BI').), 
where A' stands for the lefl regular representation of G  that is given by A() 
A(3.]ç* J,Ï)*.  will make use of this isomorphism in the discussion that follows. 
' = 3.4 3, Aut ((W*)'/A ') is isolnorphic to Aut (G 
Since A = J(A ) J and A ' 
A/(A)). Thus we obtain a hon,on,orl,hism ff'on, (A ) into Aut (G  
Bv using the isomorphism p introduced above, it is explicitly given as follows: 
u  (A (*)  0 := p- o Ad (JuJ.)]A o p. As we saw in Lelnlna 4.1 and the 
remark afler it, this holnomorphism is smjective if A a is infinite or if A is finite 
and G is finite-dimensional. But it may hot be in general. So our next goal is to 
identifv its image in detail. For this, first note that. thanks to Proposition 3.5, it is 
enough to identifv automorphisms fl  Ç(G °p such that 03 = 0 for , 
Moreover. since each 0 has the form 0 = Ad (v @ 1) for a torique v  IG(G ) with 
d = Ad v, it suces to identify unitaries v  IG(G') such that Ad (v @ 1) = 0 for 
some u e W(A ). 
Proposition 4.2. Let o be a miaimal aud int«grable action of a locall 9 compact 
quantum gro'up G = (I. A. ç, ,) ou a .factor A. 
(1) For a, 9 u  .ç(A). there ezists a uaique uaitary w(u)  IG(G) such that 
() F    (A), ,it w() i t (), , 
O, = Ad(Jw(u)J  1) = Ad(J N J)a(u)(1N u*)(J N J). 
We deaote the uitary Jw()J la IG(G') b9 v(u). 
Pro@ Let u  (W). 
(1) It is straightforward to check that a(u)(1  u*) comlnutes with any elelnent of 
the forln lça, where a  A . So it is contained in BI'Ao(COEAa)  = IC. Hence 
there exists aunitarv w  BI such that n(u)(1 )u*) = w@l. Thus a(u) = w@u. By 
applying A @ id to both sides of this identity, we obtain A(w) = w  w. Therefore 
w belongs to IG(G). So put w() := w. 
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(2) Choose/3 E 9(G °v) a.nd t, E IG(G') such that 0u = 03 and  = Ad v. Since 
G' = G °v, we have (A'(0)) = A'(v0). Therefore, by (4.3), the identitv 0 = 0O is 
equivalent to the lleXt: 
Ad ( 3& ) ( ,lO3 J  id)(U) = (.3vO&ï  id)(U) 
This is flrther equivalent to 
(,Li  &,,L)U(.L)  ,LP,L) = (3J  )U(,Li«  ). 
By using ff* = (j e &)U(J e ,L), we can rednçe the al,ove identitv to 
«(u)(1 e u*) = ,Je.] e 1. 
Consequently, we obtain «(u)= ,lv,l. 
Definition 4.3. Let /3 be an action of G on a von Nemnalm algebra P. For a. 
right «o-ideal N of G, the intermediato von Neulnam snbalgebra P(N) of P ç P 
associated to N (sec [7]) is defined by 
The fi)llowing lenllna is In'OVen in [4] for the case wh«re G is a. Woronowicz 
algebra. The clailn and its pl'oof are still valid even for a general locallv compact 
qua.nt mn group. 
Lemlna 4.4 (Proposition 3.5, [4]). Let N be a right co-ideal of G. Tken we kove 
Pro@ Denote by Nt the right-lmnd side of the al)ove clailn. Clearl) we have 
NI ç N. Let z  N. For any 9  N, p  B(H). and w  31., we have 
(p @ w)((  1)(z) = p(9(id  w)((.r))) 
= p((id @ w)((«))9)= (P @ w')((z(9  1)). 
This shows that (z) is in N1 N M. Hence (N) ç N N 31. In particular. Ar is 
also a right co-ideal of G. Therefi)re,  := [ defines a.n action  of G  on 
If z  N. then, by the above result, op(.r)  [ @ Nt. Moreover, we have 
(i«  )(;(«))= (i«  Z;)(5;('))= (Z ;  
From [16, Theorem 2.7], it follows that (z) belongs to 7(N) = (A). Thus 
.r Ç N 1 . 
Lemma 4.5. Let  be an action of G on a von Nemann algebra P. For 
intenediate von Nemnann subalgebra Q 4 pa ç p. {(id @ ')((z))  " Ç Q.  
P, }" is a right co-ideal 4G. We denote this rigkt co-ideal bg NA(Q). md call 
the right co-ideal associated to Q. 
Pro4 Let 9 e &.(Q)'. For any p,  e B(H), w e P. and ze Q. we have 
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From this, we sec that A((zd®a)(/3(x))) is inch,ded in N(Q)® M. Hence N3(Q) 
is a right co-ideal of G. [] 

Lemma 4.6. Let 13 be an action of G on, a von Neumann algebra P. |Vith the 
notation introduc«d above, we bave N = N((GOE P)(N)) for any rigbt co-ideal 
N of P. If L is the von Neumann subalgebra of   OE P generated by (P) and 
N @ C for some rigbt co-ideal N ofG p, then N(L) = N. 
Proof. Let N be a right co-ideal of p. It is plain to see that ((G P)(N)) is 
contailmd in N. Let x  N alM p  B(Hç),. Since À°P(x)  N @ M. we have 
(.,.  )= A»(.)  1 e  (g  P). 
Tlms x@ 1 belongs to (GOE P)(N). lfw  P, is a state, then 
(id @ p)(P(x)) = (id @ p@)(P(x) @ 1) 
= (idpw)(Ç(« 1))  N(( 
It follows ff'oto Lemma 4.4 that N is «ontained in OE)((G P)(A')). Therefore, we 
bave prove that N= (( OE P)(N)). 
Let L be as above. Take any st.are   A.. Then, by Lemma 4.4, we hare 
,ç = {(id e )(%)) " « e , e ,}" 
= {(id¢)(h(.) 1)« e »'. e }" 
= {(d  ¢)((,  1))' e X,e a1%}" ç (). 
In the meantime, L is clearly included in (G  OE P)(N). Hence, bv t he result of the 
previous paragraph, we get N(L) Ç N((OE P)(')) = N. 

Lelnma 4.7. Let G = (M. A. ç, ¢) be a locallg compact quantum group. Then 
1"I 9(G) = C if and only if G is coromtalive. 

Proof. If G is commutative, then we clearlv bave ]I ç(G) = C. 
Put N := IG()'. It is easy to see tha.t N is a two-sided co-ideal (von Neumann 
subalgebra) of .Î (more precisely, of ). Moreover, we have M ç(G) = /I V/N'. 
From [4, Théorèlne 3.3] (which is still valid for a locally compact quantum group), 
it follows that N =/Î if/lç(G) = c. Then Ç is cocommutative. [] 

As explained in Section 1, the mapping v  IG(G')  Ad vl ï  ç,op) is 
a topological isomorphism. Let /3,.  ç(G °p) stand for the inmge of v  IG(G') 
under this isomorphism. 

Theorem 4.8. Let c be a minimal and integrable action, of a locally compact quan- 
tum group G = (M,A,ç,,) on, a factor A. ||e set P := .Af(A) '' and define 
p 
P to be the basic extension of P C_ A, i.e, P := J, J. Itïth the isomor- 
phism p" GOEA  A, put Q := p-(P). Let w(-)'N'(A )  IG(G) and 
v(-): .N'(A )  IG(G') be the maps obtained in, Proposition .2. 
(1) The maps w(.), v(.) are continuous homomorphisros with H(A ) their ker- 
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(2) 

(3) 
Proof. 
() 

We bave 
Q! = (G OEA) {0:ue'h/(A«)} = (G OEA) {0t-(}: 
= (G  A)(ï{z( :ev()} ). 

(1) This is straightforward. 
With the original dcfinition o[ 0, ( 

E JV'(A")), we clearlv have that 

(A) {°:ueN(A)} = Pl. So we get the first equality of our assertion. Bv Propo- 
sition 4.2 (2), we llave 0, = t3,(u), which yields the second equality. The la.st 
equality follows from the fact that 0o alwavs satisfies (/3 OE id) o & = & o 0 due to 
Proposition 3.1. 
(3) This follows fronl Part (2) and Lemma 4.6. 
Let P Ç Q be an inclusion of von Nemnmm algebras. If the normalizer group 
(P) of P in Q generates Q, we sv that P is rc9ular iii 
The next theoronl is a direct generalization of [20, Theorenl 3.6], where we treated 
only the case where  is finite-dimensional. 
Theorem 4.9. Suppose that a is a minimal and inte9rable action of a locally 
compact quantum group G = ( M. , ç,  ) on a factor A. 
(1) If A' is regular in A, rb en G is cocom.mutative. 
(2) If .4  is inflnite, or if A is finite ad G is finite-dimen»ional, then the 
cocommutativity of G implies the regvlarity of A' in A. 
Pw@ Retain the notation elnployed iii Theorem 4.8. 
(1) Suppose that A  is regular in A. Then Q = (A). By Theorem 4.8 (3), we 
get 
ï {,,() . e (A)}' = {.,:(4} = C. 
In particular. M  {w(u) "   (A)} ' = C, because wc bave J3lJ = M in 
general (sec [101). Sin«e {«(u)  u e (A)} '' is a twsided co-ideal of ç, it 
follows from (2.1) that 
Hence  is cocommutative. 
(2) Assume that A  is infinite, or that A is finite and  is finite-dilnensional. 
Let v  IG('). It follows ffoln the proof of Lemma 3.5 that Ad (v OE 1) is in 
Aut ( oE A/a(A)). Bv assullption, it also follows ri'oto Proposition 4.2 and the dis- 
cussion preceding it that u e (A )  0 = Ad (v(u)OE 1) e Aut (ç A/a(A)) 
is an isomorphism. Hence there is a unique u  (A a) such that 
Ad (v OE 1) = 0 = Ad (v(u) OE 1). 
By t.he proof of Lemlna 3.5 again, we nmst have v = v(u). Therefore. the map v(-) is 
surjective, i.e., v((A)) = IG(ç'). So, if  is coconlmutative, then v((A)) ' = 
BI. Fron this, it follows that 
 {z():e(A')} = ;ï M = C. 
By Theorem 4.8 (3), Na(Q) = C. This means that Q1 = (OEA) a = a(A). 
Therefore A = P. 
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Next we would like to discuss Theoreln 4.9 (2) iii the case where A is finite and 
G is infilfite-dilnensional. 
Let F be a (countable) discrete group and  be a minimal co-action of F (i.e.. a 
niinimal action of G(F)) Oll a type II factor A. For any 7  F, define 
and call it the eigensubspace of 7- The subspaces {A(7)}er played a vital part 
in defining the Connes spectrmn F(a) of a in [18]. 
Proposition 4.10. Let F, ( and A be as above. For any 7  F, tbe eigensubspace 
A"(7 ) contains a ¢nitary. 
Pro@ By [18, Theorem 3.17], A(7)*A'(7 ) is a-weakly dense in A «, so that A(?) 
contains t)lenty of nonzer() elements for any   F. Fix an arbitrary 7  F  {e}. 
Put 
B:= ).. ",bA «, X.]'.4(7 . 
Bv using *he minimality of a and *he fac* tlmt A'  A(?) = {0} (V7 # e), one 
can easily w'riD tlmt B is a sui)factor of k&(A) = A  a&(C). Accordingly. the 
torique tracial state on A @ a&(C) restricts to that of B. So the projections 
[(1) a]' [Ô ï] 
m'e equivalent in B, since they have equal traces there. Hence there exists an 
isometry }"  A(?). Since A is finite. Y is a mfitary. 

Theorem 4.11. Suppose that c is a minimal, integrable action of a cocommutative 
locallg compact quantum 9roup G = (5I. X,,) on a II factor ,4. Then .4  is 
're9ular in A. 

Proof. Since A is finite, it follows that G nmst be of COlnpact type. Hence G has the 
form G : ((F) for a unique (countable) discrete group F. Bv Proposition 4.10. ev- 
er)" eigensubspace A'(7) contains a unitary I'(7). Cearly V(7) belongs to 
So it relnains to show that {l'(7)}er and A' together generate A. But. this follows 
flore the next two facts: (i) A is generated by {A*(7)}er: (ii) A'(-),) has the form 
A'(7) = A«I;(7) for any 7  F. [] 

Remark. We remark that, fol" a minimal, integrable action c on an infinite factor A 
with A ¢* finite, the cocommutativity of G does NOT in general imply the regularity 
of A . In fact, suppose that A is a factor of type IIIa (0 < ,k < 1). Take a faithful 
normal state w on A with  = idA, where T := -2rr/log A. We regard this inodular 
action as ail action ct of the one-dimensional torus T on A. It is well known that ct is 
a minimal (integrable) action. Note that A , the centralizer of w. is a factor of type 
Iii. Let u be in A/'(A). It is easy to see that u*cz (u) lies in (A")'çA = C. It follows 
that u belongs to some spectral subspace A(n) := {a  A : c,(a) = za (Vz  T)} 
(n  Z) of the action c. But, according to [14, Lemma 1.6], every spectral subspace 
A"(n) except A"(0) = A  contains no unit ary. This shows that A/'(A ) is contained 
in A ". Therefore A  is hot regular in A. 
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5. IEMAFIIxS ON ILP's GALOIS COFIFIESPONDENCE 

Let et be an action of a conlpact Kac algebra G = (11I, A, ç, g) on a factor ,4. 
In what follows, the action c is aSSUlned t.o ho lninilnal and imegrable, but we de 
net necessarilv assmne that ,4" is infinite. 
In [7], a complete Galois correspondelce fol a minimal action of a compact Kac 
algebra is obt.ained. According te [7, Theoreln 4.4], the map N  A(N) gives a 
one-to-one correspondelwe between the lattice of right co-ideals of G and that of 
interlnediate snbfac'tors of A  C A. 

Proposition 5.1. The inverse map of ILP's Galois correspotdetce cited above is 
given by B  N,(B). 

Pro@ Let L, .51 and 6 be as in the proof of Theol-eln 3.6. Then it is easy te check 
that. for any iltellnediate subfactor B of 4  C_ ,4 and any right co-ideal N of , 
one has 

(5.1) 

.(N) = L ® 4(N), ](L ® B) = 

(Note that ally intermediate slll)factor C of A' _C  has the foliii Ç = L ® B fol a 
Ulfique/3 as above, thanks te [7, Theorem 3.91 and (the proof of) [15. Lelllllla 2.1].) 
Therefore, by considering .21 instead of A itself if necessary, we lnay assulnc ff'oin 
the outset that A  is infinite. Then o is dominant by [17, Theoleln 2.'2] or [16. 
Proposition 6.4]. Se there exists an outer action  of Ç' on .4 ¢ such that {.4, o} 
is conjugate te {Q't< .4 `,/}. Hence we assulne that A = Q't < .4 « g:/ll(| Or = [?. 
First, by Lelnma 4.6, N = N(A(N)) for any right co-ideal N of . 
Let B be an intermediate subfactor of A' C A. Choose the unique right co-idcal 
N such that B = A(Nt) by using [7]. By the result of the preceding paragraph, we 
obtain A(N(B)) = 4(_N,(A(_N))) = A(]V) = B. This completes the proof. [] 

The following proposition is regarded as an extension of [211, Theoren 3.5], where 
we discussed the case of G being finite-dilnelsional. 

Proposition 5.2. Suppose that a is a minimal action of a compact Kac algebra 
G = (/11, A,ç,¢) on a factor A with .4 ' inflnite. Then/NÇ(JV'(A) '') = IG(G)". 
i.e., the right co-ideal erg correspondin9 te the intermediate subfactor JV'(A')" is 
Ia(C)". 
Pro@ Let P be the basic ext.elsion ofthe inclusion P := ,V'(A')" C_ 4. With p the 
isomorphisln introduced just after Lemnm 4.1, set Q := p-1 (p). Bv Lemnm 4.1 
and Theorein 4.8, we have Ï ç(ê°) = Na(Q). Frolll this and Corollary 6.3. we 
obtain Ï ç(-°) =/)(ÏN(P)' ). In other vords, ÏIG(G')' =/)(ÏN(P)' ). 
Hence, by (2.1), we have 
N.(P) = AI ç (_î f N.(P)' )' = ]Il  {¢(-Ï ç IG(G')' )' 
= M ç (M ç J IG(G')' J )'= M  (M ç IG(G)')' = IG(G)". 

Thus we are done. 

Rernark. The above proposition may be used in order te prove Theoreln 4.9 in the 
case of colnpact Kac algebra actions. 
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6. APPENDIX 

Let. a, be a minimal and integrable action of a locally compact quantmn group 
G = (M, A, , ,) on a factor A. We fix a faithflfl normal semifinite weight w on A 
and regard A as acting on H. Denote b.v 3 the modular conjugation of the dual 
weight &. The canonical implementation U of ca, associated to w is then given by 
U = 3,,(.] ® J,,). The basic extension of A 
Let. p be the isomorphism p- G,<A  A that appeared in Section 3. Identitv 
(4.2) can be rewritten in the form: 

(G.1) 

p(A(¢) ® l) : (¢® id)(U*) 

Let. A2 1)e the basic extension of A _C At. Since B(H¢)® ,4 is the basic extension of 
c(A) C G <A. the al»ove isomorphism p can be extended to the isomorphism, still 
denoted by p. from B(H)A onto A.2. Since B(H)A(CA) ' = B(H)C C, 
the equation 
defines a ,-isomorphism il ff'oto B(H«) onto 42  (A') '. Since B(H)  4  
(A)' = M'  C and G  A  (C  A«) ' = M  C, it follows from (6.1) (recall that 
(6.2) A2 n A' = H(3I'). .4, n(A")'=H(î)={(N.id)(U*).ÇM.}". 
Since kA(ç))= .lA(ç)*J = A(o R), it follows from (6.1) that 
(6.3) H((:)) = &H(z)*,L (z e lÎ). 

Lemma 6.1 (Proposition 4.4. [4]). For any interrnediate subfactor B of ,4   A. 
N.()' ®C = B(H) ® A  ()'. 
Equiva le, tlg. H (IV (B)') = A._ V B'. 

Pro@ Since B(H)®Aç(C®.4)'= B(H)®C, wesee that B(H)®Aça(B)'= 
{T ® 1: T Ç B(H), T ® 1 e c(B)' }. For T e B(H), we have 
r®lea(B)' <==> (\®¢)((r®l)a(b))=(\®¢)(a(b)(T®l)) 
( e B(H,)., ¢ e .4.) 
 \(T(id®¢)(a(b)))= \((id®¢)(a(b))T) 
(\" e B(H,)., ¢  A.). 
The last condition is equivalent to T being in N(B)'. [] 

Lemma 6.2 (Corollaire 4.5, [4]). Let B be an intermediate subfactor of.4  C_ A. 
(1) We have MCÇ(B)'CC =Q<Aça(B)' , i.e.. n(MA(B)') = AIB t. 
(2) I4 bave AÇ(B)' C = (M' C) V (Aa(B)'), i.e., A2B' = 
(A2oA') V (Al 
I 
(3) Let B1 :: JwB Jw be the ba$ic extenion of B ç A. and set 1 
Then H(R(M  AÇ(B)'))= B (A) '. In particular.  
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Pro@ (1) This follows from Lemma 6.1 and the identitv Ga<A f-/(C ® Aa) ' = 
bi®C. 
(2) This is due to the fact that Nc,(B) = Iii ç? (Iii çl Nc,(B)')'. 
(3) This follows frolll Part (1) and (6.3). [] 

Pro@ For w, choose a faithfifl normal state w0 on .4 a and put w := w0 o Ta. By [7, 
Theorem 3.9], there exists a (uuique) conditional expectation EB from A onto B. 
Let eB 1)e the Jones projection of B: eBA(a) = A(EB(a)). So BI = (AID {eB})'. 
Since eB G B1 ç (A(') ', we find that B1 = A V (B ç (A()'). From Lelmna 6.2 (3), 
it follows that / = c(A) V Ï?(Ï ç/N(B)') ® C. From Lemma 4.6, we find that 
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COMPOSITION OPERATORS ACTING ON 
HOLOMORPHIC SOBOLEV SPACES 

BOO IRIM CH()E, HYUNGWOON KOO. AND WAYNE SMITH 

ABSTRACT. \Ve study thc action of composition operators Oil Sobolev spaces 
of analytic flmctions having fractional derivatives in some weighted Bergman 
space or Hardy space on the unit disk. Criteria for when such operators are 
bounded or compact are given. In particular, we find the precise range of or- 
ders of fractional derivatives for which all composition operators are bounded 
on such spaces. Sharp results about boundedness and compactness of a com- 
position operator are also given when the inducing inap is polygonal. 

1. INTF¢ODUCTION AND STATEMENT OF IESI'LTS 
Let D be the unit disk in thc complcx plane.  shall write H(D) for the «lass 
of ail holomorphic functions on D. Let .s k 0 be a l'eal muni»er. Following [BB], we 
defilm the flactiolml del'ivative tbr J'G H(D) of ordcr  bv 
f(z) = (1 + n)az , z  D 
where 0 z is the Tavlor series of f. 
In this pal)er, we are going to investigate composition operators a«ting on holo- 
moI'phic Sobolev spaces defined in terlns of fractional derivatives. To introduce 
those hololnorphic Sobolev spa«es, let us first recall SOlne wcll-klK»Wn flmction 
spaces. For 0 < p <  and a > -1 the weighted Bergman space 4 v is the 
space of all f  H(D) for which 
= f I/(z)lp( -Izl)  dA(z) < 
,If,, 
where dA is area measure on D. Also, the Hardy space H p is the spa«e of all 
g  H(D) for which 

]o " dO 
O<r<l 

We will oit.Pli use the following notation to allow unified statements: 
AP_I = H p. 
TMs notation is justified bv the weak-star convergence of (o. + 1 )(1- [z[) a dA(z)/zr 
to dO/2zr as a -- -1. 
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Now, for p > 0, s OE 0 and et >_ -1, the hololnorphic Sobolev spa.ce A, s is 
defined to be the space of all f  H(D) for which *f  APo.  will often write 
H = A   define the norm of f  .4  by 
[[fIIAL = 
Of course, we are abusing the terre "norm", since I1 IIAe, does llot satisçv the 
t.rialagle inequality for 0 < p < 1. but in this ce (f, 9) IIf - 
 9llA. defines 
a translation-invarian metric on .4 which turns Ag into a complete topological 
vector spae. 
A fraction   H(D) that satisfies ç(D) C D induces the composition operator 
C, defined on H(D) by 
Throughout fifis paper the symbol ç will always represent a holomorphic self-lnap 
of D. In this paper we study the action of composition operators on holomorphic 
Sobolev spaces. This setting allows a unified trea.tment of composition operators 
_ 
on Hm'dy spa.ces (H p = .4 p 1.0), weighted Bergman spaces (.4 = P -1), 
and Dirichlet-type spaces 
(A,), where extensive resem'ch has already been done. 
The book [CM] is a good introduction to this work. The main results in this paper 
may be viewed as smmnarizing well-known boundedness and compactness results 
for composition operators on these spaces, and then extending them to the Sobolev 
setting. 
It is a well-known consequence of Littlewood's Subordination Principle that everv 
composition operator is bounded on A for every p > 0 and   -1: see [MS]. It is 
natural to ask how this extends to the spaces A. when s > 0. For p > 0, j > -1, 
sj  0 (j = 1.2) with a - 2 = p(s - s2), we have the following equivalence (see 
Theorem g.12 in [BB]): 
(1.1) P  A p 
AI -Sl 2,82 " 
That is, these spaces are isomorphic and have equivalent norms. In particular, when 
s < «+ we hae A s  Aa_sp. Thus it follows that eve- composition operator is 
bounded on A s when s < +t. The general situation is described in the following 
, p 
theorem. Just the statement of this and out other main results are given in this 
section. The proofs will corne later. 
Theoreln 1.1. Let p > O. s  O and   -1. 
+ ten ever composition operator s bonded on A, s. 
(a) If .s < , 
(b) Ifs- + 
p 
(i) p  2 or  = --I, then eve composition operator s bounded on A, s. 
(il) p < 2 and  > -1. ten some composition operators are hot bounded 
o 
(c) If s > + then some composition operators are hot bouded on A 
The case  = -I in part (b) corresponds to s = 0, and  previously mention 
everv, composition operator is bounded on H  = 4P-,0- The case o = -1 in part 
(c) shows that this does hot extend to H for a range of positive s, 
for the Bergnmn-Sobolev spaces. 
The boulldS on s in Theorem 1.1 can be extended when the inducing map of the 
composition operator is univalent or, more generally, of bounded valence. 



COMPOSITION OPERATORS ON SOBOLEV SPACES 2831 

The upper bound 
given in §6. We do 
sharp, but another 
be extended to the 
The equivalence 
for oel > -1 with o1 
inclusion relations: 
(1.2) 
(1.3) 

Theorem 1.2. Let p > O, s > 0 a,d a > -1. Assume that q is of bonded valence. 
(a) If p>2 and s < a+ then C is bounded on A . 
a+l 1 then C is bouttded on A,. 
(b) If p<2 ands< +, 
s < a+ inpart. (a) forp > 9 issharp: auexamplewillbe 
a+l 
hot know whethcr the ui)per bouud s <  +  in part (b) is 
examl)le will bc given that shows that the upper bound cmmot 
1)omd «+ from part (a). 
p 
(1.1) does not extend to the limiting case ee = -1. However, 
+ 1 = p(st - see), we have the folh)wing Littlcwood-Paley-type 

p < 2 === 4 p C AP_I,se, 
-- - (3i , I 
,P 
p> 2 === . -1,s2 C 
__ - Ol,S I  

hlclusion relations for different values of p are also kllown. 
aj > -1, si > 0 (j = 1,2) with +2 +ee = st - see, we have 
  P P2 

For 0 < Pi < P2, 

Let. p > 0, c >_ -1. and s >_ 0. Note that ,are have .4, C A (+2)/(+2-p) for 
ps < c + 2. as a special case of the above inclusion (s2 = 0, ni = (2)- hl case 
ps > c + 2, inclusion relations with other types of fimction spaces are known a.s 
follows: 
(1.5) 0 < ps - (( + 2) < p == -4Pa,s C As-(a+2)/p, 
(1.6) ps=c+2  4 p cVMOA. 
Here, Ae denotes thc holomorphic Lipschitz space of order e, 0 < e < 1, and 
VMOA denotes the space of holomorphic functions of vanishing meml oscillation. 
The definitions and more information on these spaces can be round in [CM] for A 
and [G] for VMOA. For details of ail the inclusions mentioned above, see Theorenl 
5.12, Theorem 5.13, and Theorenl 5.14 in [BB]. 
The boundedness (conlpactness) of a composition operator on a smaller space 
often implies the boundedness (compactness) of the operator on larger spaces. This 
general philosophy and the inclusion relations nlentioned above lead to natural con- 
jectures. The nlethods developed below in §2 to addrcss tllese conjectures require 
some restriction on the parmneters. In particular, the case (1.2) is left open since 
out nethods do not apply when the target space is a Hardy-Sobolev spa.ce. 
Theorem 1.3. Let X C I" be any of the inclusion relation in (1.3) - (1.5), and 
assume for iclusion (1.3) that see < 1. a, for mchtsio,(1.4) that c2 > -1 and 
s2< 1+(1+c2)/P2. 
(a) If C,  X --, X is bounded, the Cv  }" --, Y is bounded. 
(b) If C " X  X is compact, then C " Y - Y is compact. 
Inclusio,1 (1.6) was left out of the preceding tlleo,'eln, but we have the following 
partial result in that case. 
Theorem 1.4. Let p > O, o >_ -1. s > 0 and assume ps = a + 2. 
(a) If C, " A, ---, AP, is bounded, then C,  VMOA -- VMOA is bounded. 
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VIIOA is compact. 

\Ve also mention the elementary inclusion relations that. for ail p > 0, s >_ 0. 
c _> -1, and e _> 0, 

,tP P 

In §3 we will give a result analogous to Theolem 1.3 for these inclusions, but with 
SOllle restrictions on the parameters. 
As a first application of Theoreln 1.3. notice that it can be used to prove the 
case c > -1, p > 2. and s = +__! in Theorem 1.1. Then H p CAPa s bv (1.3), and 
so everv COlnposition operator is bomded on AP, bv Theorem 1.3. hl the other 
direction, once criteria for Co to be bOulded or compact on the larger spaces are 
known. Theoreln 1.3 can be used to l)rovide necessary conditions fol bouldedlleSS 
or COlnpactness of Co on the slnaller spaces. For example, bv taking Ae as the 
larger space, we bave the flllowing consequence, which has |)een kllown for p _> 2 
(Theorem 4.13 in [CI[]), while it bas been known to be false for p = 1 (p. 193 in 
[Chi]). So. the gap l < p < 2 is now filled in. A more general version is proved as 
Theorem 3.3 below. 

Theorem 1.5. Let p > 1 aTtd suppos« C " II' -- Hï is bo«vded. Then the angular 
derivative of ç exists at all points ç  i)D whcre the radial limit (Ç) of ç exists 

A basic problem in the studv of conlposition operators is to relate function- 
theoretic properties of ç to operator-theoretic properties of the restriction of C'o to 
 4 p bv 
various spaces, as in Theoreln 1.5. When ps < a + 1. we have .4Pa s "  a-sp , 
(1.1), and criteria for (7  4 pi  4 p= to be bounded or COlnpact are klloWll. The 
 O1  2 
characterization is that a generalized Nevanlinna colnting function for , satisfies a 
growth condition if p2 >_ pi, or an integrability condition if p2 < p: sec [Sml] and 
[SY]. The results in [Sml] and [SY] do hOt apply when ps >_ a + 1 in either the 
domain or the target space. |11 that case, criteria in the form of Carleson lneasure 
conditions for a measure defined using a lnodified comt.ing fimct Joli call be obt ained 
as in Theorem 2.6 below, with sonle restrictions on the paralneters ctj. pj, and si. 
This Carleson-type criteria in Theoreln 2.6 will be used to prove Theorelns 1.2 and 
1.3. \Ve also lnention that for the special case p = 2 other teclmiques m'e available. 
since the llOrlll of a ftlllctioll in A, lnaV be given in terlns of its power series 
coeflïcients. These spa.ces are examples of what are called weighted Hardv spaces 
in [CM], which is a good reference for colnpositiol operators acting on these spaces. 
Characterizing when a colnposition opera.tor is bounded on HsP, s > 0, seelns 
much harder. The diflïicultv is that (1.1) does not provide ail isonlorphisnl with 
a space of functions defined with full derivatives, and the llethods used to prove 
Theorem 2.6 do hot apply. We have froln Theoreln 1.1 that. for anv p > 0 and 
s > 0. there exists a functiol ç such that C is hot bounded on Hs p. A positive 
result is that C is COlnpact on certain Hs p whelever ç is of bounded valence and 
ç(D) is COlltained in a polygonal region COlltained in D. This is the special case 
pi = P2 of the following result. Fol" a polygon P inscribed in the trait circle, let 
O(P) denote 1/rr times the measure of the largest vertex angle of P. 
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Theoreln 1.6. Let 1»2 _> p, > 0 and as.saine 0 <_ s < min{&p2, }'1 Let ç be a 
h, olomorphic flo.**ctio' of bo'uded vo, lence tat:ig D ito a polygon P iscribed i 
th u, it circle. IfO(P) < p(1--sp2) thf @ " Hf   Hy 2 is çom]ioct. 
p2 (1- sp ) " 
When s = 0 and p = P2, this has long been known: see [ST]. When s = 0 and 
p  p, this result is basicallv contained in [Sml]. These results (when s = 0) do 
hot require the hypothesis of bounded valence.  will prove a more general result 
in Theorem 5.5. 
In the next section we develop the change of variable methods that we use to 
study composition operators, which we then use to give Carleson measure-type 
criteria for these operators to bc bounded or COmlmct. These criteria are then used 
in 3 to prove Theorem 1.3. Next. in 4. the proofs of Theorem 1.1 and Theorem 1.2 
e given. Simple geometric criteria are then developed in 5 for loundedness and 
compactness of a composition operator between holomorphic Sobolev spaces when 
the inducing map is polygonal. The pal»er concludes, in 6. with several examples 
which denlonstrate that out theorems are sharp. 

2. ]A('KGttOUND: CARLESON-TSt'E CRITERIA 
Oto" al)proach t.o studying COlnposition operators on the spaces A, involves a 
change of variable ri'oto z to w = ç(z). The equivalen«e (1.1) allows us to assume 
that s is an integer, and then standard non-retiraient change of variable methods 
can be applied. This gets quite compli«ated when s is an integer greater than 1. 
Thus, for simplicity and claritv of presentation, we confine our attention to the case 
,s = 1. Thisenables us to cover parmnetersp.  and s with +(1-s)p > -1 
by using the equivalence A,  A2+(l_)p.1 ri'oto (1.1). The change of variable 
method for s = 1 is summarized as follows. 
For a hololnorphic lnap   D  D and w G D. define the modified counting 
function Ç,(ç, w)corresponding to the measure (1- [z[)dA(z) by 
oEç,(ç, w) =  Iç'(z)lP-(1 -Izl)  
where the sure is over the set {z  ç(z) = w}. As usual, the zeros ofç- w are 
repeated according to their multiplicity. The change of variable formula we need 
ses the lllestlre 
Then. bv the area formula (see Theorem 2.32 in [CM]), we have the following «halage 
of variable fornmla. 

Proposition 2.1. Let p > 0 atd c > -1. Then, we bave 

[(f ° ç)'(z)[P( 1 --Izl2)dA(z) = / [f'(w)[Pd#, (w) 
 . (D) 

for functions f  H(D). 

Note that Proposition 2.1 cannot be directly applied to the case s = 1, because 
Ttf(z) = f(z)+zf'(z) by our definition. This difficultv is overcome bv the following 
proposition. 'e will off.en write X < }" if X _< C" for some positive constant C 
dependent only on allowed parmneters, and X  }" if X < }- < X. 
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Proposition 2.2. Let p > O. c >_ -I and a E D. 
ri. we bave 

for f e H(D). 

Then, for every positive integer 

n--1 

Pro@ We prove the proposition for a = 0. The proof for general a is sinfilar. The 
eq**ivalece IIIIA.  2=0 Ilfl)llA is proved in Thorem 5.3 of [BB]. Thus, 
llfl A > n--1 
--0 If()(°)l ÷ IIf()llAa is clear by subharmonicity. 
Now, we prove the other direction of the inequalities. Since H(D) is dense in ail 
holomorphic Sobolev spaces bv Lemma 5.2 of [BB], it is suflîcient to show that 
(2.1) IIfllA < If(0)l ÷ IIf'llA, f  H(D). 
First, assume either ve > -1 or 0 < p _< 1. Let. f  H(D). For each/3 > -1, we 
bave bv Theorem 1.9 of [BD], 
1 /D Tf(w)Gl(z)(1 -Iwl=)aA(u') 
f ( ) : - . 

where 

1{1 
Ge(Z)=z (1-z)+ 1 . 
Therefore, choosing ./3 > -1 sufficiently large, we have by Lenmm 4.1 of [BB] 
(ve > -1 or 0 < p _< 1 is used here). 
(2.2) IIfllA < IIfllA+  IIfllAX+ ÷ IIf'll o+ 
A p . 
It is easy to see tlmt, given g > 0, there exist a constant C > 0 and a COlnpact 
subset Iç= {z  Dlz ] _< r< 1} ofD su«h that 
[If[l£+ < ellfllA ÷ C sup If(z)l. 
Taking  > 0 sufiîciently small, we have by (2.2), 
IlfllA¢ £ IIf'llA+ ÷ sup If()l 
< If(0)l + IIf'llA + sup If(z) -- f(0)l 
< If(0)l ÷ IIf'llA¢ ÷ sup If'(z)l. 
Since supe/,- If'(z)l £ IIf'llAg by the subharmonicity of If'l p, we obtain (2.1) as 
desired. 
Now, consider the case  = -1 and p > 1. Note that 
[f(e °) - f(0)l < If'(te°)ldt. 
Therefore, by Minkowski's inequality, we bave 
[If- f(0)[[H <  [f'(te°)[PdO dt <_ IIf'llH». 
which implies (2.1). The proof is complete. [] 
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Having seen Proposition 2.1 and Proposition 2.2, it is now clear that the behavior 
of C, when the target space is A p depends on that of the measure ttp, For 
botm(ledness and compactness of C, the criteria for It, turn out to be Carleson- 
type conditions in certain cases. To pi'ove it, we ueed a couple of lemnms. 

Lemma 2.3. A bounded subset of any of the spoces APo, s. 
a >_ - 1, is a normal fam ily. 

where p > O. s >_ O. and 

Pro@ First assume o > --1. Using (1.1), a bounded set X in AP.s is also bounded 
in some A p where  is a nonnegative integer. Recall that there is a constant 
such that 

I.q(,*')l 
for ail g  A (see, fol" example, Tlmoreln 7.2.5 in [R1]). By Prol)osition 2.2, this 
shows that the functioIs in X are mfiforlnly bomMed on COlnpact snt)sets of D. 
Hence X is a normal family. The proof of the resull fi)l'n = --1 is sinfilar, since 
.4ç,, æ c .4ûy æ by (1.4). The l,roof is COlnplete. 

In the next lelmna we will need the estimale that if a > -1 and/3 > 0, then 

fD (1--11)" dA()  1 
(2.3) Il - zl ++ 

A reference is Theorem 1.7 of [HKZ]. 

Lemma 2.4. Let p > O. c > -1 and s > O. Let N > +- - s. Put g(z) = 
(1 - zO) -v for a, z  D. Then. we have 
IlgallAs,.  (1 -I1)--+,   D 
where the constants in this estimate depend on N, s. a. and p, but are mdependent 
ofa. 

Pro@ First, assume c > -1. Let k be the smallest integer satisfying k _> s. Then, 
we have AP,  APa+(k_s)p.k by (1.1). Thus, by Proposition 2.2, we bave 
- { fD (l _ D[)'+(k-)PA(z) } /P [ça AP "" 1 + Z cN'j[a[J + CN'k]a[k 11 - zg[P(N+k) 
where c,j = N(N + 1)...(N + j- 1). Thus, by (2.3), we have 

]g AP  

where C = C(N,s,c.p). The desired estinmte follows. Next, assume c = -1. 
API A p 2p 
Note that o,a/p+s C _, C Ao.  by (1.4) and thus 
Ilgoll A,:,..," £ IlgoIIA,.., £ Ilgoll,«/=,:,.:,/,+.,  
O11 the other hand, we have 

II.qll.g.,,.,  ( -I,1) --÷  
by what we have just proved for the case ct > -1. This completes the proof. 
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For anv arc I C BD define the Carleson square over I tobe 
SI = {re ie  1 -II] < r< 1. e e  I}, 
where [1[ is 1/(2rr) rimes the Euclidean length of I. Also, let ô denote the complex 
differential operator, i.e., ôf = f' for f  H(D). 
The next lemma asserts that certain operators are compact.  review the 
definition, since when p < 1 the spaces involved are hot Banach spaces. Suppose 
X and }" are complete topological vectors spaces whose topologies are induced bv 
metrics. A contilmous linear operator T " X  }" is said to be compact if the image 
of everv bounded set in X is relatively colnpact in } Due to the metric topology of 
}" T will be compact if and onlv if the image of every bounded sequence in X bas 
a subsequence that converges in }'. Also, linearity of T allows us to only consider 
sequences in the mfit ball of X. 
In the following lemma, part (a) is well known: see Theorems 2.2 and 3.1 in IL1]. 
Part (b) is certainly known to experts. For example, the case k = 0, p = q, and 
a > -1 ocçurs as Theorem 4.3 in [MS]. A proof is included here since we do hot 
know a reference. In out application, we will take k  1. 
Lemma 2.5. Assume that one of the followin9 three conditions holds: 
(i) a>-l.O<pGq; (ii) a=-l,p=qk2: (iii)«=-l.0<p<q. 
Let k be a nonne9ative ite9er and p be a positive finite Borel measure on D. 
(a) ô " " A  Lq(d) is bounded if and only if 
O (]IIq+q(«+)/v) . I C 0D. 
I+(SI) 
(b) 0  . A  Lq(dp) i+ complet « nd ol if 
c+.> C+> = + 
O. 
Moover. the norm q the nap in (a) satisfies the inequality 11011 +  CIlll, 
IIl  tSe supremum of the quanlity <S0/llq+q< +2v o,,e I c OD. 
Proof.  provide a proof of {b). ç first prove the suciellCV. So, assunle 
and let {f.. be a bounded sequcnce in A, say of norm at lnost 1/2. We must 
show that {f} contains a subsequence whose k-th derivatives converge in Lq(d}. 
ecall that we have observed that a bounded set in A is a normal falnily, and 
so bv subtracting the limit function and re-indexing an appropriate subsequence, 
We may assaille that 11511A¢  1 and that {f} and hence {f t converges to 0 
uniformly on compact subsets of D. We need to show that {f[)} converges to 0 
in Lq(dp). Let e > 0 and write 
liN& IIL.(d.) = Ifa)[qdp + Ifa)[qdp, 0 < r < 1. 
The first tenn is easilv handled. For any fixed r  (0.1), the uniform convergence 
of {f)} to 0 on rD allows us to find N(r) such that 
< 
D 
Turning to the second terre, by hypothesis we can choose r = r  (0, 1) so that 
thc mesure dr(w) = XDD(W)dp(w) satisfics v(SI)  zI] , whenever [I]  1- r. 



COMPOSITION OPERATORS ON SOBOLEV SPACES 2837 

where fl =/,'q + q(2 + o)/p. For II1 > 1 - r, we subdivide I into m arcs of length at 
lnost l-r, where 'm 5 [I[/(1-r)+ 1  2Il[/(i-r), and observe that SI(DrD) 
is contained in the Carleson squares associated with the smaller arcs. Thus. the 
previous estimate shows that 
oe(S/)  e(1 - r)   2ellI  
in this cse as well. Note that we used  2 1, which is a consequence of the 
hypotheses, for the last inequality. Thus, xve sec rioin (a) that there is a constant 
Ç such that 
Slll) 
Combilmd with the previous estimate, this shows that IIf} ) I](d,)  0 as required. 
Now, wc prove the necessity. Suppose (2.4) is false. Tlmn there exist a constant 
C2 > 0 and a sequcnce of arcs I, C 0D such that I, I  {1 and 
(2.5) (SI,,)  clI,l +(+/. 
Let  = lIl and ( Ç OD be the center of I, fi»r each . Fix a large integer 
N > (+ 2)/p. Let g,(z) = (1 - (1 - )zÇ,) -N and put f, = gllgA. - Note 
that [10,,llg  a;Np++z by Lemma 2.4. Tlms, {f} converges uniformlv to 0 on 
compact subsets of D. Now, using the COlnpactness of 0 k  A  Lq(dp), pick a 
subsequence of {f,, } whose k-th derivatives converge to 0 in Lq(dl t) and use the 
saine notation for that subsequence. Note that l1 - (1 - a)zÇ]   for z e SI,, 
and n large. Thus, by (2.5), we bave 
(a)  
for ail large z. This is a contradiction, because IIf, I1)  0. The proof is 
complete. 
Now, a change of variables md standard m'gumelts give us the following Car- 
leson measure characterizations of 1)oundedness and compactness. As discussed in 
the first paragraph of ihis section, we restrict out considera(ion of the orders of 
differentia¢ion to certain rmlges: mmlvsis of the general case seems too complicated 
for this paper.  also lnention again that when 8p ( ç 1 or p  2, other methods 
are available and much more is known: see the discussion following Theorem 1.5 in 
the Introduction. 

Theorem 2.6. Assume ttat one of the Jbllowing three conditions holds: 
(i) O' 1 > -1.0 < Pl _ P2: 
(il) a = -1,pl = Pz >_ 2; 
(iii) cri = -1.0 < Pi < 
Also. as.sume cz >-1 ad 

o 1 ÷ 2 1 a2 + 1 
(2.6) 0_<s < 1+ , 0_<s <l+-- 
Pi P p 
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 A v is bounded if and on lg if 
(a) c A2,.,--, , 
(_.,) p,+(l_)p2(SI) : 0 [I[ (2+)p2/p1+(1-s1)p2  I C 
(b) C  A p  A p is compact if and only if 
181 282 
P p2,a2+(1-s2)P2 " 
Pro@ Here, for brevity, we prove the suciency for boundedness and the necessitv 
for compactness. The other ilnplications can 1)e seen 1)y ey lnodifications. Also, 
let p  for silnl)licity. 
: Pp2,a2+(1-s2)p2 
First. we provc the siCiCllCy for bomdedness. One may easily modify the proof 
for conq)actness. So, suppose that p satisfies (2.7). 
Note that 
1)y the first part of (2.6). Thus, by Lelnma 2.5 (a) (k : 0), we have 
for fimctions g holomorphic on D. 
Also. note that a2 + (1 - s)p2 > -1 1)y th(' second part of (2.6). It follows flore 
(I.I), Proposition 2.2. Proposition 2.1 and (2.9) that 
P2 
5 [f(ç(0))l + IIf'll_+(+.l»«,)_. . 
Now, bv Proposition 2.2 and (1.1) again, we see that the sure in the last line above 
is equivalent to 

" P2-- +( al)P2/Pl 
Next, it is clear that [R s' f(0)[  IIfl[.4,Sl- Also, it is easy to verify using Lemma 
2.5 (a) (k = 1) that 
< I I/IIA«EE = II/IIA:,I 
II0 '/IIAX_+,+I»» '  
Putting these estinlates together, we conclude the boundedness of C " 4 pi 
- 1,81 
 2,2 " 
A p is compact. Suppose that (2.8) does not hold. Then there exist a constant 
C > 0 and a seqllenCe of arcs In C OD Sllc that IIl  0 and 
(S[n) OE Cin[ (2+eI)p2/pI+(1-sI)p2 
Let  = ILI and Ç  0D be the center of I for each n. Fixa large integer X > 
(a+2)/p s. Lette(z) (1 (1 )zç) - and put f 
..... IiIIAx.I" Note 
[[ff[IA,l   -81+(2+1)/1 by LIIII OE,. TlllI, {A } COllVFge unifoFllll}" tO 
0 on colnpt subsets of D. Therefore, using the colnpactness of 
4 p2 and 
APa,s=, we mav. pick a subsequence of {f o ç} that converges to 0 in . . 
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use the saine notation for that subsequence. Now, first using Proposition 2.1 and 
then proceeding as in the proof of Lenlnla 2.5, we have 
IIf ° llAo,s  IIf  I[A= 
2+(l--s2)P2,1 
 5 -(+«)/'. Il-(1 
>C 
for all large n. This is a contradiction, because IIf o IIA=.  0. The proof is 
complete. 

3. FROM SMALL SPACES TO LARGER SPACES 
%'e now turn t.o the proof of Theorem 1.3. For convenien«e we divide the theorem 
into more easily managed pieces, considering ea«h implication separately as well 
boundedness and conlpactness. 
Theorem 3.1. Let pi, si and j (j = 1,2) be as in tbe bypotbeses of Theorem 
2.6. In addition, assume tbat  -  = si - s2. 
Pl P2 
(a) C A m v (compact. resp.) if and onl'y if C is 
 ,  A., is bounded 
bounded (compact, resp.) on A 
2,8 2  
(b) If C9 is bounded (compact, resp.) on 
Proof. Note that (2 + a)p2/pt + (1 - 81)P2 = 2 + 2 + (l -- 82)P2- Thus, (a) follows 
v= by (1.4). Thus, (b) follows from 
from Theorem 2.6. Also, noie that 4 ma,s C Aa=,s = 
(a). 
It is straightforward to check that when a = -1 and pi = P2  2. the hypotheses 
(2.6) in Theorem 2.6 are equivalent to s2 < 1 in (1.3). Thus, Theorem 1.3 with 
inclusion (1.3) is an immediate consequcnce of Theorem 3.1. Similarly, when 
-1 and 0 < p < p2, the hypotheses (2.6) in Theorem 2.6 are equivalent to a2 > -1 
and s2 < 1 + (1 + a2)/p2 in (1.4), and so Theorem 1.3 with inclusion (1.4) follows. 
The proof of the next theorem uses some properties of the pseudo-hyperbolic 
distance p on D. Recall that the pseudo-hyperbolic distance between points a and 
b in D is given by 

a -- b 
p(,b)= ï-2 I 

We use D(a, r) to denote the pseudo-hyperbolic disk of radius r and center o. Recall 
also the well-known and useful identity 

- b 12 (1 -112)(1 -Ibl 2) 

a,bD. 

In particular, it is a consequence of this that 
(3.1) I 1 - bl  1 -I12  1 - I12 
whenever b  D(a, 1/2). 
The next result covers the inclusion (1.5) in Theorem 1.3. and so completes its 
proof. 
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Theorem 3.2. Let p > O, o > -1 and a+_A2 < s < 1+ a+e If ('o is bounded 
-- p P 
(compact, resp.) on 
Pro@  first prove the assertion on boundcdness with the additional assumpt ions 
that o > -1, p > 1 and «+ < s < 1+ «+ Note that a+(1-s)p> -1 and 
P P 
a p bv (1.1). Choose a  D such that ]ç(a)[ > 1/2, and 
therefore A, m . a+(1--s)p.1 - -- 
consider the test function foc(z) = log(1 -ç(a)z). Then, by Proposition 2.2. we 
have 
/ Iç(a)l Zl2)a+(1 
[IL  çll:+,l_.,.., E I - ç(a)ç(z)  Iç'(z)l( -I -'}PdA(z) 
- 
For z  D(a, 1/2), we have 1 -]z 2  1 -la z, by (3.D. Also, the Schwarz-Pick 
Lemma tells us that ç(z) 6 D(ç(a), 1/2), and so 1 -ç(a)ç(z)[  1 -Iç(a)  from 
(3.1). Using these estimates in the lt terre in the display above shows that 
(3.2) > 
For the last inequality we used that D(a, 1/2) contains a Euclidean disk with center 
a and radius comparable to 1 - ]a 2, and that ç'l p is subhannonic. 
Meanwhile, since f(0) = 0, we have 
(3.3) 
where the lt. equivalence holds by (2.3), because sp > a + 2. Putting these 
estimates together with the assumption that C " A,  A, is bounded, we get 
aDSUp 'p'(a)[ { (1-- 'ç(a)[) } -l+'-(a+)/v(l - 
This is equivalent to the bomdedness of C on A_(«+)/v; see [Mai or Theorem 
4.9 in [CM]. 
Now, consider the general case o > -1 and a+ < s < 1 + a+. Choose q > p 
 p P 
so large that q > 1 and s < 1 + a+ . Put 13 - (a+2)q 2. Then.  > a > -1 
P q P 
and +2 -- +2 Now, bv (1.4) and (1.5), we have 
q P 
4 
Also, note that + < s < 1+ + Now, suppose that C  A,» + A is 
q q  
bounded. Then C " A,  A, is bounded by Theorem 3.1 and thus so is 
C " As-(a+2)/p  As-(a+2)/p by the result for the special case we proved first. 
This proves the assertion on boundedness. 
ç now prove the assertion on compactness. Note that As_(a+2)/p and A. 
are MSbius invariant, in the sense that every composition operator induced bv a 
conformal automorphism of the unit disk lna.ps each space into it.self, and contained 
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in the disk algebra of holomorl»hic fimctions on the unit disk that extend tobe 
continuous on the closed disk. Thus a general theorem of 3. H. Shapiro [Sh] asserts 
that compactness of C' on each of these spaçes implies that ç(D) is a relatively 
compact, suhset, of D. We recall also that 
If(0)l + ,,p{( -]:loe)t-lf'(z)l  z  D} 
is an equivalent norm on Az; sec Thcorem 4.1 in [CM]. Now, let {f} lw a l»oundcd 
sequence in A.«-(a+2)/v-  must show that some subsequence of {f,ç} converges 
in A-(+2)/v-  know tllat {./} is a normal falnily, and thus a sut)sequence 
(which we still call {f}) converges to some f G H(D) tmiformly on compact 
subsets of D. Also, if C is compa.ct on A,, then it 
4 v C A-(a+2)/p. HellCe (1 --I=l=)-+(+=)/vl'(z)l i ll, liforllllv bO, ll,ded o11 D. 
and it follows that 
I (0) -   (0)l + (1 -Izloe)l-++oe)/l () - ,f'  ()ll'()l 
uniformly on D as n  , sincc ç(D) is c'ontaincd in a compact subsct of B. This 
means that {f, o F} c«mvcrgcs to thc fim('tion 9 = .f o ç in A-(a+2)/v, and so 
C  A-(a+2)/v  A-(a+2)/v is compact. Thc 1)roof 
Criteria fi)r C to lc 1)omded or compact on A: are known. So Thcorem 3.2 can 
be used to provide necessary condit.ions for boun(ledness or compactness of Cç on 
the smallcr spaces. In particular, we recall that the bomdedncss on A: implics the 
existence of the angular derivative of ç at all points of the unit circle whcre ç has 
a. radial limit of modulus 1: ste Corollarv 4.10 in [CM]. This l)roves the following 
theorem. 
Theorem 3.3. Let 1) > O.  > -1 atd + < 
 p P  , 
is bouned, the the a9ular derivtttive of ç exists af all points Ç  OD where the 
ro.dial limit ç(ç) of ç ezist.s a,,d so.ti.sfies I(çl = 1. 
As mentioned in the introduction, the conclusion of Theorem 3.3 is false for 
 = -1, s = 1 and p = 1. Thus, for  = -1, the lower bouud 1/p for s camaot be 
decreased in general.  also give an example which shows that the lower bound 
s > a+ in Theorem 3.3 is sharp in case o > -1. See Examl)le 6.3 below. 
The proof of the next theorem is based on Theorem 2.6. So, for simt)licity, we 
restrict our consideration to the orders of differentiat.ion covered there. 
Theorem 3.4. Let p > O. a > -1. s > 0 ad 
 p 
(a) If 1 + + s > e > 0 a'nd C is bouded (compact. resp.) on A,+«. tbe 
p 
so is C on A,. 
Proq[. Let I be an arc in the unit crcle, and let ç(z) = w G SI. A standard 
argument, using the Schwarz Lemma then tells us that 1 -]z  1 -[w]  I[, and 
SO 
w  SI. Hence 
Iz,a+«+(-)v(SI)  Ille#,+(i_,)v(SI)  
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and statelnent (b) is now an imluediate consequence of Theorem 2.6. The proof of 
(a) is similar alld will be onfitted. [] 

We finish this section by giving the proof of Theorem 1.4 from the iltroduction. 
which we rest.ate for convenience. 

Theorem 3.5. Let p > O, o >_ -1, s >_ 0 and aasume ps = a + 2. 
(a) If C " AVa,s - A, s is bousd«d, thes C " VMOA - VMOA is bounded. 
(b) If ç is univalent and C " A, s -- 4v,s is compact, then C  VlklOA -- 
VIklOA is compact. 

III the proof below and elsewhere, we use the notation dist(a. OE) for the Eu- 
clidean distance between a point a and the bounda.ry of a set E. 

Proof. If C is bolu,ded Oll .4P,s, then ffoin (1.6) we bave that ç = Cz  4P,s C 
VlklOA. Also, it is easy to see that (7, is bomlded on VlklOA if and onlv if ç  
Vlk[OA: see, for exalnple, [Sm2]. This gives part (a). 
For the proof of (b), we recall that when ç is Ulfivalent. C is COlnpact on VMOA 
if and Olfly if 

dist (w, 0ç(D)) \(Dl(W) 
(3.4) lira = 0: 
,,,-+,- (1- [wl) 

see Theoreln 4.1 in [Sln2]. Also, it is ait easv cousequence of the Koebe distortion 
theol'em that if ç is univalent, then 

(3.5) 

( -l[2)l'(=)[  dist(ç(z),Oç(D)), z  D: 

see Corollary 1.4 iii [P]. 
First, consider the case p > 1 and c > -1. With ps = e+ 2 < p+ct + 1, case (i) 
of Theoreln 2.6 (b) tells us that C is COlnpact on AP,» if and only if/xp,p_2(SI) = 
o(1I[ ) as [I[  0. We prove part (b) by showing that this fails when C is hot 
compact on VMOA. From (3.4), if (7 is hot compact on VMOA, then there is 
an g > 0 and a seqllence {Wh} C (D) with [zt,,[ -- 1 and dist(w,,0ç(D)) > 
e(1 -Iwl). Let I,, be the arc of the unit cirçle with ceuter u,,/[w,[ and length 
[I,[ = 2(1 -[w,[). Since ç is univalent. 

L {Iç'(-')l(1 Izl2)F-2d.4(w), 
]'P'P-2(Sln) = In -- 

where w = ç0(z). Froln (3.5), Iç'(z)l(1 -Iz[ 2)  (1 -Iw,l) for w in the disk with 
center w, and radius e(1 -Iwl)/2. which yields the lower bound 

 (s) # o(llI"), III - 0. as desired. 
Hence [Lp,p_ 2 
Now, consider the general case p > 0, c _> -1 and suppose that 
is compacL With ps =  + 2, choose q as in the proof of Theorem 3.2. That is, 
choose q > p so large that q > 1 and put fl : sq- 2 > -1. Then +2 _ +2 = s, 
q p 
and so A, s C A, s by (1.4). Thus. froln Theorem 3.1 we see that C. " A, s  A, s 
is compact and thus so is C  VMOA  VMOA by the result for the special case 
that we have proved above. The proof is complete. 
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4. COMPOSITION OPERATORS ON AP,s 
In this section we prove Theorems 1.1 and 1.2 flore the introduction. For con- 
venience, we divide these results into more easilv managed pieces. As meutioned 
in the introduction, it is well known that every composition operator is bounded 
. 4 p bv (1.1), 
on A for ail p > 0 and  > -1 Note that we have A, 
when sp <  + 1. Thus, it follows that cvery composition operator is boundcd on 
A, s whenever sp <  + 1. Thc next two theorems complete the description of the 
general situation, as statcd in Thcoron 1.1. 
Theorem 4.1. Letp>O, s>0 ad >-1. Ifs- a+l atd 
(a) p > 9 or = -1 then every composilimt operator is bonded on 4 v ; 
(b) p < 2 and a > -1. then some composition operators are uot bounded on 
Pro@ If  = -1. then s = 0 and so every composition ol)erator is b(mnded on 
A, = H v. If a > -1. p  2 and ps = a + 1. then froln (1.3) we have that H p C 
A,« Hence part (a) follows ff'oto Thcorem 1.3. since all composition operators are 
bounded on H p. 
a+l 
Turning to thc proof of (b), firsI note that A, s  Ap_l, 1 p 
P 
P is necessarv for @ to be bounded on Ap_l. 1. Thus it 
Also, ç = @z ¢ Ap_l, 1 . 
suces to show that if p < 2 there is a bounded analytic function F 
Ap_13. The 
case p = 1 of this statemeut is outlined in exercise 9(a) iii Chapter VI of [G]. That 
construction can be modified to work for p < 2. For completeness, we sketch the 
argument. 
Let p < 2 and consid«r the flmction 
f(z) =  k-llçz 2. 
Since the series for f is lacunary with square summable coecients, it is known 
that f  BMOA. This is an easy consequence of BMOA being the dual of H 1 
together with Paley's Inequality for the coccients of an H 1 flmction ([D], p. 104). 
or see [Mi] for another approach to the proof. Next, it is easy to verify that if 
:¢A,,.={wcD" 1-2 - 
then [ff(z)[  -l/P2. This leads to the approximation 
£ if,(z)lp( 1 _ izl2F«dA(z )  1 
flore which we see that f  
Av_I. 1. This is hot the l'equired example, however. 
since f is hot bounded. But since f ¢ BMOA, there are bounded flmctions Ul 
and u2 on the unit circle such that ef = Ul + fi2 where ri2 denotes the harmonic 
conjugate of u2. Here, we are using the saine notation for a boundary flmction 
and its harlnOlfiC extension. Then If'lp  IVu I p + Iv.u21 p by the Cauchy-Riemama 
uations, and it follows that there is a bounded real flmction u on the circle such 

that 
L iv,()i(1- 
Now let F = exp(u + i'ù), so that F 
Ap_,, and the 

lzl)P- dA(z) = oo. 
is a bounded analytic flmction satisfying 
proof is complete. [] 
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c+l 
The proof of the next theorem, covering the case s > ---, reqnires two lemmas, 
which will a.lso be used in the next section. 
Lelnnla 4.2. Let p > O. s  O. and a  -1. Then the following inclusions hold: 
(a) A,. c A ç for e > 0: 
(b)  A . for O < < p. 
Ap+a_e,l+ s C ., 
Moreover, both iwlu.s'ions are bomMed. 
We remark in I)assing lhat, when s = e = 0. a = --1 and p  2. the inclusion 
iu (a.) holds and this is just a restatement of the well-known Littlewood-Paley 
inequalily. When s = e = 0, o = -1 and 1 < p  2, the inclusion in (b) holds and 
lhis is a restalement of the (lual of the Litllewnod-Paley inequalily. 

Proof. Bv definition of the holomorphic Sobolev spaces, it is suflïcient to consider 
the case s = 0. First. consider lhe case o > -1. Then, xve have 

il lld 

.4 c 4 ' , P 
o+e Ap+a+e,l,  > 0 

4 v " A; 4 p  APo, 0 < e < p 
" p+o--e,1  o,e/p C. oe+e,e/p 
where the eqnivalences are ri'oto (1.1) and lhe inçlusions are clearlv bounded. 
Now, assmne (t = -l. Let J'G H p and put 
1 
(4.) .x(f.,.) = . If(,.d°)FdO. o <  < 1. 
Then, for anv  > 0. we have Mp(f.r) = O ((1 - r)-), which implies Mp(f',r) = 
O ((1 - r) -t-) (see Theorem 5.5 of [D]). Thus. for e > 0. integralion using polar 
coordinates shows lhat 
fb If'(z)lP(1- 'z]2)P-l+edA(z)  '(1- 
F)--(1+6)P+P-I+dF. 
With  small enough so that pO < e, this integral is convergent, and so (a) holds. 
Now, assume 0 < ¢ < p. Çonsider the case p > 1 first. Let p' be the conjugate 
exI)oncnt of p. Bv the fundamental theorem of calculus and H61der's inequality, we 
have 
  ,{1  
[, II(e '°) - I(0)F"0  ,. II'(rd°)[& eO 
JD 

xvhere 

It follows that 

I fo I , "1 p/p' 
C = (1 - r) -p (P-a-)/Pdr < 

IIf[Iï4,, £ [f(o)l p +/D If'(:)lP(1 -- 
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and the Saille is true for p = 1 by a trivial lllodification of this argunlent. This 
proves (b) fol" p _> 1 by Proposition 2.2. When 0 < p < 1, we bave tlle inclusions 
Ap/(l-e) _ Hp/(1-e) Hp 
A;_ l_e, 1 C -1,0 -- C 
wheïe the fiïst inclusion conles ri'oin (1.4). This colnpletes the proof. [] 

We next show that certain inçhlsions t)etwoen hololnOl-l»hic Sobolev spa«es are 
conlpact. 

Lemma 4.3. Let p > O. s >_ O, ¢ > 0 and c >_ -1. Then the followin 9 itclusions 
are compact: 
p 4v A v 
A,,s+e C C 
Proof. We first consider the case  > -1. Let {f} be 
4 p is compact, we nmst show that 
A, s. To show that the inclusion A,, C . 
some subsequence of {f,} converges in 4 p I1 is well known tlmt the inclusion 
P 
P is compact: ste, for examplc [Sml]. Thus there exists g 
,4 C . 
subsequenc of {f } (whic for convenienc we continue t o denote {f })suc that 
sfn  g ill .4 pt+e. Now. chooscb  (D) Sllch that h = g. It is then clear that 
4 p This completes the proof that ,4 s C 4 p is 
h Ç 4P +e,s and f  h in . ,+e,s. , 
compact. Now, since we have by (1.1), 
4 , p  Ap 
" a,s+e C Aa+pe,s+e . 
and the first inclusion is compact, we con«huh' lhe compactness of the inclusion 
A p 
Now, assulne  = -1. Choose positive nunibers 1. 2 such that 0= < a < 
min(p, pe). Then, we have the relations 
4 v  v v A v 
Al,s+ e C « p-l+pe-6,l+s+e Ap-l-6A+s Q Ap-l-6,l+s 
The first, and la.st inclusions are bounded froni Lemma 4.2. the equivalence is ffOln 
(1.1), while the relnaining inchlsion is compact hy the previously established part 
of this lemma. Hence AP -1,s+e C AP -1,s is çOlnimct. For the compactness of A p -l.s C 
A p noe that 
-l+e,s 
4 p A p  A p 4 p 
 -ls C p-l+e/2.1+s -l+e/2,s C -l+e,s" 
The first inclusion is bounded from Lemma 4.2.  the isomorphism ri'oto (1.1), 
while the renmining inclusion is compact bv a previously established palt of this 
lemlna for a > -1. Hence AP -1,s C AP -+e,s is compact. The proof is complete. 

Theorem 4.4. Let p > 0 and o 
operators are hot bounded on AP,,. 

> -1. Ifs > +1 
-- p 

then some composition 

Proof. Since a = Csoz Ç A, s is necessarv, for C to be bounded on -4P,s, it suflïces 
to show that if s > +__1, then H  \ AP, -J= 0, where H  denotes the class of ail 
p 
bounded hololnorphic filnctions on D. Suppose to the contrary that H  C APo.,. 
Then this inclusion map is continuous bv the Closed Graph Theoreln. while the 
inclusion map AP,s C Aï.s was shown to be COlnpact whclmver a < /3 in Lemma 
4.3. The hypothesized lower bound for s now allows us to choose  _> 1 such 
that c +  < .sp <_ c +  + 1. Moreover, we can choose « > 0 so Sllmll tllat 
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P HP/(«+a+l+«-sp) from (1.4). 
0 < c + 6 + 1 + e - sp < 1, which gives Aa+a_l+e, , C 
Consequently, we bave a chain of inclusions 

tt  C APa  C A p C tt p/(a++l+e-sp) 
, a+-- 1-t-e,s " 

Thus the inc|usion H'° C H p/(a+a+l+e-sp) can be viewed as a product of a compact 
nmp and two bounded maps, and hence is compact. But {z  } is a bounded sequence 
iu H  for which no subsequence converges in H Pl(a+'++e-p). This contradicts 
the COlnpactness of the illclusion lnap, aud the proof is colnplete. [] 

The next theoreln (also stated as Theorem 1.2 in the introduction) shows that 
the upper bounds for s in Theorens 4.1 and 4.4 can be extended whell the symbol 
ç' of the COlnposition operator is of boumled valence. 

Theoreln 4.5. Let (, > -1 and let ç be of bounded valence. Assume that either 
a+l 
0_< s _< «+--v if p_>2. or'O_< s < --K- +½ if O < p < 2. Then Co zs bounded on 

Pro@ \Ve use the Carleson measure criteria frOlll Theoreln 2.6. We need to esti- 
llla[ e 
P,a+(1-s)p(qI)=i Z I'(z)]P-2(1-]z]2)«+O-S)PdA(w) 
o(z)=w 
for arbitrary arcs I C OD. 
First., consider the case p > 2. By aSSUluption we have sp _< a + 2. Note that, 
since ç is of bounded valence, there is a Ulfifornfly bounded lmmber of terlns in the 
sure inside the integra| above. Next. we set w = ç(z) and use the Schwarz-Pick 
Lemlna, which asserts that ]ç'(z)] < (1- ]w[2)/(1- [z]2), and then the elelnentary 
inequality 1 -[zl 2 <_ C(1 -[wl 2) to get that 

which from Theorem 2.6 is equivalent to boundedness of C on APa, . We uote that 
the hypothesis p _> 2 was used in getting the first inequa|ity, and sp _< c + 2 was 
used iii the second inequa|ity. 
Now, consider the case p < 2. What we have now is sp < c + 1 + -. By the area 
formula (Theorem 2.32 iu [C[]), we have 

f 
IAP,a+(1-s)P° (SI) = Jsal-' (si) Iç'(z)[P(1 -Izl2)+O->dA(z) 

Note that, since ç is of bounded valecce, we bave 

Æ I'(z)[2dA(z) = £ Z l dA(w) £ I,I 2 
-(s,} , :e_{} 
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where the first ilmquality is provided by H/31der's inequality. To estilnate the integral 
above, recall that everv composition operator C is bounded on A, 13 > -1, and 
that this is equivalent to 
(4.3) __ £_1(SI)( 1 __ iz[Z)ZdA(z) 
for all I C 0D: sec section 4 in [hIS]. Silwe, by hypothesis, 2( + (1 - s)p)/(2 - p) 
> --1, we Call combine the estinmtes iii (4.2) and (4.3) to get that 
lZ,a+(l_s)p(i) 
From Theorem 2.6. this is equivalent to C being bounded on A. s. 
complete. 

The proof is 
[] 

Our final result in this section shows that the conlpactness of the inclusions in 
Lemma 4.3 extends to ail colnposition operators with a certain restriction on s. 

Theorem 4.6. Let p > O, a >_ -1 and s > O. A,ssume that either (i) o. > -1, 
0 <_ s < c+___ilp or (ii) c = -1. s = 0 or (iii)  = c+.__!p .p>_2. 
(a) (Tv A,,+« -- Ag,, is compact. 
(b) c  
 Aa_e, * -- APa,, is compact whenever ce - e  -1. 

. 4  
Pro@ We may view the action of Co - a,s+e -- APa,, as follows: 

p Ap c 
- A,. 
Aa,s+ e C a,s 
Also, if o - e > -1 then we mav view the action of C  A p  .4 s as follows: 
p C 4p 
A_«,, c A,  
The ilmlusions above are both compact by Lemma 4.3. and everv composition 
operator is bounded on A, by Theoreln 1.1. Thus, both operators m'e compact. 
The proof is complete. 

Carleson lneasure criteria for Co to be bounded or COlnpact Oll Hï are known; 
see Theorems 4.11 and 4.12 in [CM]. For s hot an integer, characterizing when 
Cso is bounded or compact on H seems much harder than the analogous problems 
on the Berglnan spaces. The problem is that for the Hardy spaces, (1.1) does hot 
provide isomorphisms with spaces defined using fldl derivatives. Thus. we are led 
to the following. 

Problem. Characterize ç for which C is bounded (compact) 011 HPs , 8 > O. 
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5. (',OMPOSITION WlTH A POLYGONAL ,IAP 

Here. xve find simple criteria for the COlnpactness of the composition operators 
betv«een holomorphic Sobolev spaces induced by polygonal lnaps. 
Recall that dist(a.0E) denotes the Euclidean distance between a point a and 
the boundary of a set E. 

Lelllllla 5.1. Suppose that P is a polygon inscribed in the unit circle with a verrez 
al v and let rq(t,) be the vertex angle al v. Then. given a Riemann map  of D 
onto P. there es:.isls a neighborhood N of v such that 

I¢(z)l  ( -Iqo(z)l) -1/'(), 
( -14)  (1 -Iq(z)l) /)-I dist(v)(z). OP) 

for all z  ç-(N). 

Proof. Reca.ll that ç extends to a holneOlUorphisn of D outo P (see, for exalnple, 
Theoreln 14.19 of [R2]). Assulne v = 1 aud ç(1) = 1 for simplicity. Also, let 
q = q(v). Then, a reflection argmnent yiehis 

l l  (): C ( 1 -- Z )  + 0(I I I +',) 

for SOlllt, constant c ¢ 0 and for ail z near 1. Thus, we have 

Iv/()l  I: - l ":  I: -v)(z)l -I/  (1 -Iv)(z)l) 

for z near 1. The last equivalence in the disploEv above holds, because ç(z) is 
contained in a lontangential region with the vertex at 1. This proves the first 
equivalence of the lemlna. The second equivalence is now a consequence of the 
estimates 
(11 _]z[)]ç'(z)]_< dist(ç(z).OP)<_(1-[z])lç'(z)[, 
which hold since ç is mfivalent: see Corollary 1.4 of [PI. The proof is complete. [] 

Lemma 5.2. Let P be a pollgon inscribed in the unit circle. Assume b > -1 and 
a + b > -2. Then, there exists a constant C > 0 such that 

/oe 1-lwl) a dist(w, iï)P)bdA(w) <_ CIII 
çS I 

for all arcs I C OD. 

Pro@ Let us introduce a temporary notatiolL For an arc I C 0D with center at 
ç  0D and III = 25, we let S5(ç) = SI. 
Assmne that  is sufficiently small and P çl S5(ç) ¢ . Then. there is a constant 
C. depending only on P, such that S5(ç) c Sc5(v) for some vertex v of P. Assume 
t, = 1 for simplicity. Assuming that  is sufficientlv small so that Se, e(1) contains 
no vertex of P other than 1. note that 1 -lu, I  Il - u,] for w G PO Sc6(1). Now. 
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we bave 

(1 -[w])  dist(tv, OP)bdA(w) 

constant depelMing ouly on P. The estiluate for large 

[] 

Recall that D(z. 1/2) deuotes tlw pseudohyi»erl»olic disk. Let D(z) = D(z. 1/2). 
In the following we let dA,(z) = (1 -Iz]=)dA(z) for c > -1. The following lenmm 
is proved fol" a = 0 in IL2], and the saine proof works for general a. 

Lemma 5.3. Let c > -I and It be a positive finite Borel measure on D. Assume 
p > q > (I. Then, there is a constant C such that 
if and onlv if r G L(A) where r(z) = u(D()) 
A,,(D(z))" 
For a polygon P inscribed in the uuit circle, recall that O(P) denotes 1/ times 
the measure of the largest vertex angle of P. 

Proposition 5.4. Let pi > O. ai > -1. si >_ 0 (j = 1.2) and assume 
O1 3-2 1 a2+l 
(5.1) s < 1 + , s < -- 
Pi P P2 
Let  be a holomorphic function takin9 D into a pol19on P inscribed in the unit 
circle. If 
pl (O2 -[- 2 -- 82102) 
(5.2) 0(P) < 
P2(al + 2 -- SlPl)' 
 4 1 -- A, , is bounded. 
lhen C¢ " Ol,Sl 2, 2 
Mo.reover, for functions ç of bounded valence, the second part of (5.1) can be 
replaced by the weaker condition that 
c + 2 c2 + 1 1 
(5.3) s2 _< -- if p2 >_ 2, or s2 < -- +  if 0 < P2 < 2. 
P2 P2 - 
h either case the eq,uality tan be allowed in (5.2) for P2 >_ Pl. 
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Pro@ Let ç0 be a Rielnann nlapping of D ollto P all([ put ¢ = ç- o ç. Then 
ç = ç0 o  and Oms C¢ = Ç¢oO = Cç'C¢o. Note that CO " 4 pz  4 p 
is bomded by Throrem 1.I or Theorem 1.2. This shows that we only need to 
prove the proposition for ç = ç0- So, in the rest of the proof, we assume that ç 
is a iemann map of D onto P. For simplicity, let  = pj + j - sjpj and let 
?=2+j-jpj forj=l,2. 
First, consider the case p2 OE Pl- By Theorem 2.6, ve need to show that 
(5.4) t'p, (SI) = 0([II (2+)p/m+(1-)p) 
for ail arcs I. As in the proof of Lemnm 5.2, we onlv need to considcr I centered at 
a vcrtcx of P for which ]I] is sufficiontly small. Given such I, we havc bv Lemma 
5.1 and Lomma 5.2, 
, ]'(-i(w))]P-2( 1 -]-1 (u,)]2)OdA(u,) 
 [ dist(w,0P)u(1 -]w[)(/°-i)ud.4(w ) 
IP 
where 0 = O(P). In he last inequality we used the fact that ff2 > -1, 2  0 from 
(5.3) and tlms 
Thus, we have (0.4) by (5.2) and (5.5). Also, the smne proof works in case the 
equality holds in (5.2). 
Next, consider the case p < p.  lnav, assume ç(0) = 0. Let f  ,4 m, be 
an arbitrary fimction such that f(0) = 0. Since p < p, we have  > -1 bv the 
first part of (5.1). Thus, by (1.1) and Proposition 2.2, we lmve 
[[f][A,  [ f'(w)[m(1 --[w[)dA(w)  
Also, we have by (1.1), Proposition 2.2 and Lemma 5.1, 
Now, define mesures 
dpa(w) = (1 -[l)dA(), 

and let 

r(z) = p2(D(z)) z  D. 
pl(D(z))" 
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By Lelmna 5.3, we need to show that v E LP(p) where p = Pl/(Pl - P2). Note 
that/t2(D(z)) = 0 if z is outside of some t)olygonal region Q. ()n the other hand, 
for z G Q. we have 
p,(D(z))  (1 -Izl) »l+, 
pee(D())  (1 -Izl2) +(/e-1)+ee', 
the first esçimate is standard and the second one çall be verified with (5.6) bv 
modiÇving the proof of Lomlna 5.2. According]y, we lmve 
(=)  ( -i=l)=-»,+,/e-,,b( ). 
[t fo]lows that r e L(pt) if and on]y if p[fl2 - fl + (1/8 - 1)72] + fit > -2, which 
urns out to be thc saine as (5.2). his completes the proof. 
Theorem 5.5. Lct pj > O. ej  -1, si  0 (j = 1,2) and assume (5.1) holds. Let 
ç b« a holomorphic function taking D into a polflgon P iscrih«d in th¢ unit circle. 
If (5.2) holds, fh«n Ç" 4 m  4 p= is compact. 
Mor«ov«r. for flmctio,s ç of bovnded valence, t« second part of (5.1) can be 
r«plac«d by the w«aker co,ditio** (5.3). 
In Examplc 6.1 below, we show that (5.3) providcs the sharp upper bound of 
+2 for s2 when p2 > 2. Whilc we do hot know whcther it docs the saine when 
P2 ( 2, the ni)per bomd for s2 when p2 ( 2 cannot be extended to :+ as is 
P2 
shown by Example 6.2. Nevcrthelcss, Examplc6.4 shows the upper bomd of 
in (5.2) is sharp in eithcr case. 
Pro@ Assume that (5.2) hokls and choosc ¢ > 0 suciently small so that (5.2) 
holds with a + e in place of  Bv Lemma 4.3 we have 4 p Ç 4 P and 
. 4 m  4P2 
the inclusion is compact. Thus, it is sufficicnt to show that C 
is bounded. In case au > -1. we sec that C¢  A',+«,  -4Pa,s is bounded bv 
Proposition 5.4. 
So. assume 2 = -1. Note that with a2 = -1 there is no s2 satisfying the 
second part of (5.1). Thus. we only need to be concerned about the case where 
is of bomded valence and (5.3) holds. First. consider the case p2  2. In this case, 
we can view the action of @ as follows: 
Ara cv p 4p  
a+e,s  Ap2-1,se+l Ç ' -1,s 
where @  Ama+e,s  APp-,s+ is bomded bv. ProI)osition 5.4 and the inelusion 
cornes from (1.2). Therefore, @  A m  4 p is boCmded. Next, consider 
lE,81 " 2,8 2 
+2 _ 
çhecasep2>2. Choosepe(2,p2) ande >-1. Also, leçs=  +s2 . 
Then. we tan view the action of C  follows: 
4m c ApÇ.s  4p  
where C " A m 
+e,s  A,s; is bounded by Proposition 5.4 and the inclusion 
cornes from (1.4). The proof is complete. 
Remarks. 1. As mentioned in t.he proof above, t.here is no s2 satisfying the second 
part of (5.1) in case a2 = -1. Thus, we have no conchlsion in Theorem 5.5 for 
general ç in ce the target space is a Hardy-Sobolev space. 
2. Note that the condition (5.2) holds vacuously if a+2 +2 
P P2 
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6. EXAMPLES 
rVe llOW give sevcral exmnples delnonstrating that our theorelns are sharp. For 
tiret purpose we introduce the so-called lens maps. Fol" 0 < 11 < 1 we denote by çn 
the flmction defined bv 

er(z) ' - 1 

(6.1) qa,(z) -- er(z)' + 1" z C D 
where er(z) = (1 + z)/(1 - z). Let n(D) = L. Then, çn is the Rielnann map 
of D onto the sui»set L of D bounded by arcs of circles lneeting at z = +1 at an 
angle of lift, and fixing the points -1, 0, and 1. Because of the shape of the range 
L,» such a map is called a "lens map". Note that L is contailmd in a polygon 
inscribed in the unit cir«le. 
By a straightforward calculation, we bave 
(6.'-,)  -I()1  1 - zl ', I'(z)l  I] - zl 

for z near 1. 
The first example shows that the upper bomul s < +___2 in Theorem 1.2(a) is 
 p 
slmrp. Also, this exalnple shows that thc upI)cr bound so < az+ in Theoreln 5.5 
is sharp when p2 k 2 and  is of bounded valence. 
Example 6.1. Let 1 »> 1, o >-1 and + < s < 1 + a+l Thon, thereexists a 
P P 
P 
lens map n  -4,s" In particular, C is hot bounded on Aa, s. 
Proof. Choose 0 < 1 < 1 sucientlv small so that sp k IP + a + 2 and consider 
the corresponding lens lllap ç Note that A   A P bv (1 1). Therefore, 
" ,  a+(l--s)p,1 - " 
we bave bv Proposition 2.2 and (6.2). 
 ./ I'o (z)lp(1 -Izlu)+(1-PdA(z) 
aD 

Note that (r/- 1)p + er + ( 1 - s)p <_ -2. because sp _> 'lP + er + 9. Thus, an obvious 
estinmte iii an angle with vertex at 1 shows tiret the last integral above diverges, 
as desired. [] 

1 
We do hot know whether the upper bound s < a+__! + 7 in Theoreln 1.2(b) is 
sharp. However. the next exalnple shows t.hat the upper bound cmmot be extended 
 1 
to + as in Theorem 1.2(a.). Also. this is related to the assumption, s., < =+ + , 
p  p2 
in Theorem 5.5 when P2 < 2. 
Example 6.2. For each p  [1,2), there exist  > -1, 0 < s < 1 with s < + 
 p 
and a univalent holoInorphic self-map  of D such tiret   A p In particular, 
C is hot bounded on A,,. 
Pro@ P..lones and N. Makarov have showll (see Theorem D(2) in [aMI) that for 
any p < 2, there exist a univalent holomorphic self-map çp of D and a constant 
' satisD 
c > 0 such that the integral ieans of p . 
P--I+c(2--P)   P  __ 
(6.3) (1 - ,') - I (wp- ") > 1 
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for some sequence 7", -- 1. Here, we are using the notation introduced in (4.1). 
Note that, for anv f E. a, /3 > -1, we bave 

a.ild so 

1 
(1 - r)+SIP(f,r)= (/3+ 1)/lç(f, r) .f (1 -t)dt 
< (/3+ 1) lllP(f,t)(1-t)13dt=o(1) 
! 
as r  1. This, together with (6.3), yields çp tg Ap_2+c(2_p)2. hl other words, we 
have çp tg Ap_2+c(2_p)2,1 p bv. Proposition 2.2. N-te that thc hvpohesis p _> 1 is 
used here to ass,re that p - 2 + c(2 - p) > - 1. Now, «hoose s E [0, 1) su«h that 
sp - 2 + c(2 - p)Z > -1 and put « = sp - 2 + c(2 - p)2. Th('u, wc bave s < +2 
Also. since 4 p 
 p-2+c(2-p)=,l "-4Pa,s bv. (1.1), we havc çp  4 v,s. [] 

The next examplc shows that the lower bomd s > +-- in Thcorem 3.3 is sharp 
when o > -1. 

Exanlple 6.3. Let p > 1, ,. > -1 and put s -- +2 Then. there exists a 
P 
holomorphi« self-nmp ç of D with ç(1) = 1 su«h that C'  APo, s -- APa,, is b(mnd('d 
but ç does hot have angula.r derivative a.t z = 1. 

Pro@ Let ç = çn be any lens lnap. Note that a + (1 - s)p > -1. Thus, as in the 
proof of Proposition 5.4, we have 
m that @ " A,  A,, is bo,mded bv Theorem 2.60). Clearly, ç does hot have 
an angular dcrivative a.t 1. 

The next example shows that the upper bound for O(P) in Theorem 5.5 is sharp 
when Cil > -- 1. 

Example 6.4. 
O I > --1 and 

Let pi, si, aj be as in the hypotheses of Theorem 5.5. Assume 

Then, f o çu  4 2 

PI (02 + 2 -- 
(6.4) p2(o + 2 - SlPl) < ri < 1. 
for some f G 4 P 
 Cq ,S I " 

Pro@ Let ç = ç,. Choose 0 < a < 1 such that ]ç(a)l _> 1/2. Also. by using (6.4), 
choose « > 0 sufficiently small so that 

(6.5) (2 + o2)/p 2 -- .» + a 
- <]<1 
(2 + O1)/p -- S 

Now, consider the test fimction f(z) = log(1 -ç(a)z). Let k _> s be a positive 
integer. Then we have 4 pI  A pi 
« C,,I..S1 tl_]_(__S1)p,] by (1.1). Therefore, bv Proposition 
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2.2, (2.3), (6.2) and (6.5), 
(6.6) < 
On the other hand, for e2 > -1, 

and thus by (6.2), 
(6.7) 

(1 - I(a)l 2 ) 
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DISTRIBUTIONS OF CORANK 1 
AND THEIR CHARACTERISTIC VECTOR FIELDS 

B. JAKUDCZYK AND M. ZHITOMIRSKII 

ABSTRACT. oEre prove that any l-parameter family of corank 1 distributions (or 
Pfaff equations) on a compact manifold 5I ' is trivializable, i.e., transfornlable 
to a constant family by a fanily of diffeonmrphisms, if ail distributions of the 
family have the saine characteristic line field. The characteristic line field is a 
field of tangent lines which is invariantly assigned to a corank one distribution, 
If is defined on lI n, if , = 2/,', or on a subset of AI" called the Mmtinet 
tlypersurface, if n : 2k -I- 1. Our second main result states tllat if two corank 
one distributions have the saille characteristic line field and are close to each 
other, then they are equivalent via a diffeOlnorphisln. This holds tmder a weak 
assumption on thc singularities of the distributions. The second result ilnplies 
that the abnorlnal curves of a distributiol determine the equivalence class of 
the distribution, among distributions close to a given one. 

O. INTRODUCTION 
The well-known Gray theoreln [G] states that any 1-parameter fmnilv of contact 
structures on a compact manifold ,l).a+l is trivializable, i.e., transformable to a 
constant falnily by a fmnily of diffeomorl)hislnS. Our first main result generalizes 
this theorem to he case of 'singular contact structures", for which the contact 
condition is satisfied on a dense subset of M, and to cora|lk one distributions 
on llmnifolds of even dilnension. In these cases the famil.v of distributions has to 
preserve, when the parameter changes, a characteristic line fiel(l. The characteristic 
lie field is a field of tangem lines which is invariantly assigned to a corank one 
distribution (it is defined on M  if  = 2/,', or on a hypersm'face of/1I  if  = 2k+l). 
Our second main result states that if îwo corank Olle distritmtions have the Saille characteristic line field raid are close to each other, then they are equivalent via a 
diffeomorphism. It lneans, in particular, that the chm'acteristic line field contains 
COlnplete information about the geometry of singularities of the distribution. Our 
results hold under a weak &ssumption on the distributions, called conditiol 
saying that the depth of a characteristic ideal of the distribution is nondegenerate 
at singular points of the characteristic line field. 
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Let us state out second result in the simple case where the characteristic line field 
does not have singularities. Assume  = 2k >_ 4 and consider a smooth differential 
1-forln ce on /1I n. Let (w A (dw)k-1)(p)  0 for ail p  /1I n. Such a forln w defines 
the distribution A= kerce, called the quasi-contact structure defined by w. Then 
dw(p), ïestricted to the distribution Ap) = kerw(p), is of nmximal possible tank 
2k - 2 aud has 1-dimensional keïnel. We define the characteristic line at p by 
L v = ker dw(p) 
The characteristic line fleld L = L(w) is the fiekl of characteristic lines p -=, L v 
on /11. The folloxving fact is a special case of our Theoreln 1.2, the case where 
singularities are absent. (It is also a special case of a theorem in [MZh], Appendix 
A, concerning corank 1 distributions of constallt class.) 
Theorem 0.1. Let A be tbe quasi-contact structure on a compact orietable man- 
ifold III 2 defined by a 1-form w. and let & be a 1-form such that L(&) = L(w). If 
& is sujïjïciently C2-close to ce, then there exists a diffeomorphism of M 2 sending 
ï = ker& fo A. 
If a compact orientable manifold ]I 2a adnfits a quasi-contact structure defined 
bv a global 1-form w, then its characteristic line field is generated by a nonvanish- 
ing global vector field and the Euler chmacteristic of/I 2a is eqnal to zero. Thus 
a manifold /I 2a with uonzero Euler dmracteristic adnfits only corank one distri- 
butious with singular characteristic line field. Eveu if/I 2a admits a quasi-contact 
structure, siugularities nmv appear naturally when restricting a corank one distri- 
bution to a subnmnifold of even dimensiou. Therefore, it is natural to ask whether 
Theorem 0.1 holds in the preseuce of singularities. Theorem 1.2 in Section 1 gives 
a positive answer under assumption (A), saying that the singularities of the char- 
acteristic vector field bave a natural dcpth (and codimensiou, in the analvtic case). 
This assumption excludes singularities of iufinite codimension. 
To state a siufilar result in the case n : 2k + 1. we introduce the set of points 
where w does hot satisfy the contact conditiou: 
S = {p  IIl 2+  (w A (dw)k)(p = O}. 
This set is called the Martinet hypersurface. The Martinet hypersurface is the set 
of zeros of the flmctiou 
H = ce A 
where {2 is a volulne forln. If S is elnpty, i.e., ce is a contact 1-fonn on 3I 2+, 
then A = kerw is globally equivalent to any distribution , sufficiently close to 
A. This follows frolu the theoïem of Gray lnentioned above. Assume now that 
S is louempty. \Ve shall call A = kerce a Martinet distribution if it satisfies the 
following two conditions: 
(a) dH(p) ¢ 0 for all p  S (then S is smooth), and 
(b) As = kerces is a quasi-contact structure on S, where ws is the pullback of 
wtoS. 
At each point of S we can define the characteristic line 
Lp = kerdws(P)]ks(p), p  S. 
The characteristic line field L = L(w) for a Martinet distribution & is the field of 
tangent lines p -- Lp on S. It has no singularities. The following fact is a special 
case of out Theorem 1.4. 
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Theorem 0.2. Let A = kerw be a Martinet distribution on a compact orientable 
manifold 1112k+1 . If the Martinet hypersurfaces ad the characteristic line fields of 
A and/ = kera5 are the saine and & is sujïficietly Ca-close to w, t]en there exists 
a diffeornorphisrn of tle rnanifold sending A to A. 
Martinet distributions fonn a restrictive class of corank one distributions (L may 
have singularities). In particulm, any Martinet distribution on M 3 has the Mmtinet 
hypersurface S, which consists of two-dimcnsional tori (each connected component 
of S has zero Euler characteristic, since S admits a l-dimensional foliation defined 
by L). Out Theorem 1.4 will generalize Theorem 0.2 to thc case of glmral corank 
one distributions and PfaIf equations on/ll 2k+1. 
The appemauce of the characteristic line field L as one of the main invariants 
of corank one distributions A has the following history. The first to study L -cas 
J. Martinet in [Mari for the simplest occurring singularities of X on IR 2+ (when 
L has no siugularities). J. Martiuet also started to study typical singularities of 
L in the 3-dimensional case. These singularities were roughly classified in [JP], 
where the existence of a modulus in the classification of characteristic line fields 
was shown. It was proved in [Zhl] by obtaining a normal form for A that this 
modulus is the only iuvariaut of A. This gave thc complete local classification of 
germs of generic 2-distributious ou 3-manifolds, with the characteristic line field as 
the complete invariant. 
In the book [Zh2] the second author gave a classification of finitely determined 
singularities of corauk one distributious and PfaIf equations on mauifolds of any 
dimension. In this case again the chm'acteristic line field L is a complete invariaut. 
This justifies, to a large extent, lnaking the followiug 
Conjecture. In the space of germs at 0  IR ' of corank one distribations on IR ', 
there is an ezceptional set of infinite codirnension such that for any two distributions 
Ao and As away front this set, with the saine characteristic line field L, there ezists 
a local diffeomorphism op : (IR ', O) -- (IR ', O) redacing X to A0. 
In a weaker form, with n = 3, this conjecture has already appeared iii [JP] and in 
a letter from J. Mmtinet to the second author (in 1989). In this case the conjecture 
tan be deduced from the results in [JZh2], [JZhl]. The exceptional set consists of 
germs that do hot satisfy the assumption (A) or that do not have the property of 
zeros (sec Section 1). In [JZh2] we proved that away rioto the exceptional set the 
restriction of the distribution to the Martinet hypersurface is a complete invariant 
for any n = 2k + 1. If k = 1, then the restriction eau be identified with the 
characteristic line field. 
The results of the present paper concern global corank one distributions close to 
a fixed distribution. They also deal with families of distributions. Iii this setting we 
eliminate some of the diftàculties iii the above conjecture which are due to non-close 
germs and the necessity of preserving a fixed point (the source of the germ). 
Our transition to the global approach was inspired by the results in [Gol] and 
[MZh, Appendix AI. In [Gol] it is proved that a fmnily of Engel structures Et on 
a 4-manifold is trivializable provided that the characteristic line field of Et does 
not depend ou t. The result in Appendix A of [MZh] states that two close global 
corank one distributions of any constant class (iu the Cartau-Frobenius sense) are 
diffeomorphic provided that they have diffeomorphic characteristic foliations. This 
is a generalization of the Gray theorem. All these results apply to objects without 
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singularities, whereas in the present paper we allow any singularities except certain 
ones of infinite codimension, excluded by the assumption (A). The presence of 
singularities leads to the main ditïficulties iii ollr proofs. In Section 2 we explain 
tllat, the assumption (A) is natural and give examples showing that it caimot be 
weakened. 
In [JZhl] and [JZh2] we ohtained local realization theorems {for gerlns at a fixed 
point }, t heorems characterizing the set of all possible characteristic line fields if r = 
3. Combined with the reduction theorems, thev lead to a number of applications 
including classification results. In this paper we leave aside the diflïcult task of 
obtailfing global realization theorems. The absence of such theorems restricts, at 
present, the possibility of drawing imlnediate iuteresting conclusions concerning 
global classification of corank one distrilmtions. 
Note that the assmnption of closeness which appears in Theorems 0.1 and 0.2 
{and later in Theorelns 1.2 and 1.4} is essential for out results. Already two contact 
structures that are far ff'oto each other are, in general, hot equivalent. A classifica- 
tion of contact structures is known only on certain 3-diluensional manifolds, sec e.g. 
[TEG]. We hope that our results can be used in glolml contact or quasi-contact ge- 
»metrv for studying singularities of 1-forms which appear when two global contact 
(or quasi-contact) structures are joilmd by a path. 
There are natural consequences of our results concerning characteristic curves of 
a distrilmtion, also called sigular curves or abormal cu.rves in sub-Rielnannian 
geometly, all(l t]le geolnetry of distril,utions (cf. lAI, [BH], [LS], [Mon]). These 
curves coincide in out case with t]le integral curves of the characteristic line field. 
Under thc assumptions of Theorerns 0.1 ad 0.2, as well as those of Theorems 
1.2 and 1. it Se«tiot 1. the singular curves of a «otanC" 1 distribution &termine 
the equivalece class of the distribution, amog ditributions close fo a gi-ven one 
it the C  topolog. 

1. TATEMENT OF tlESULTS 

We will deal with Pfaff equations, v«hich are more general objects than (coori- 
ented) corank one distributions. Let. M u &,note a compact, orientable, Hausdorff 
lnanifold of dimension n _> 3. Bv definition, a Pfaff equation is a set of differential 
l-ri»tins on M' generated, as a module over the ring of fimctions, bv a single 1-form 
w. In other v«ords, a Pfaff equation is a 1-form on .1I ' defined up to lnultiplica- 
tion bv a nonvanishing flmction. We dcnote the Pfaff equation by P = (w). If w 
vanishes at no points of M', then (w) can be identified with the field of kernels of 
w--a coorientable hyperI)lalm fie]d in TIIl '. In general, a Pfaff equation is a more 
general object, siuce we do not exclude the possibility of w vanishing at some points 
of the manifold. 
Ail objects in this paper will belong to a fixed category which is either C  or 
real analytic C ". 
The case  = 2/,'. To any Pfaff equation P. and in particular to any cooriented 
corank Olle distribution, one can associate the characteristic line field. 

Definition 1.1. If n is even. n = 2/,', then anv vector field X defined by the relation 

XJ = to A (dw) k-l, 
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where t is a.lV vohune form, a.ll(l ce any geuerator of P, is called a cbaracteristic 
vector field of P. The line field on In generated by X, i.e., the lnapping 
is called the «haracteristic l{e fie[d. Thc characteristic line field will be denoted 
by L or L(w) or L(P}. The set {p  I   X(p) = 0} is callcd thc set of singular 
points of L and dcnoted by Sing(L). 
It is easv to check that anv two characteristic vector fields differ bv multipli- 
cation by a nonvanishing fuucIi(m, and consequently thc characteristic line field is 
invariant.ly related to the Pfaff equation P, i.e.. the choice of the generator  of P 
and the volmne form [ is irrelevant. Note that if ( A ()k-1)(p)  0. then the 
definitions of Lp and L coinçide with those given in the introduction. 
In the presence of singularities we need ail invariant that describes a "degree of 
degencration'" of singular points of a characteristic vector field X (for n odd il was 
introduced in [JZh2]). First we define i/il invariant that is slightly stronger than 
the set Sing(L). 
Let p  Sittg(L ). Thc loc«d chwracteristic ideal Ip at a point p G .I 2k of a Pfaff 
equation P is the idcal Iv(X ) in the ring of functioli gerlns at p. gcnerated Iv the 
coefficients (i 1 ..... a,, of a characteristi« vector field X of P. in some cooMinate 
svstem near p. It is easv to sec that the ideal Ip is invarialitlv related to the gerln 
at p of P = () (the choices of a chara«teristic vector field and of a local coordinate 
system are irrelevant). The germ at p of the set Si,(L) is the zero set of 
Definition 1.2. If n = 2k and p  Sittg(L), then wc define dp(P) = dp(X) as 
«(P) = d,,pth IdX). 
Recall that the depth of a proper ideal I C R of a ring R is the maximal length 
of a regular sequence of elements in I. A sequence ai ..... af  I is called regular if 
al is hot a zero divisor in and. for any i = 2,..., r. the element ai is hot a zero 
divisor in the quotient ring R/(a,.... ai-l), where (al ..... (ri-l) denotes the ideal 
generated bv ai,..., (i-1. Bv definition, depth R = . 
Remark. In the alialvtic category, dp(P) is equal to the codimelIsion in Ç' of the 
germ at p of the set of complex zeros of the ideal Ip (i.e., the zero level set of 
the ideal generated by the complexification of the generators of Ip in some local 
coordilmtes). This follows ff'oto the fact that the complexification does hot change 
the depth of an idem of analytic fimctioli germs (cf. e.g. [E]) and ffoto the equality 
of the depth(I) and the codimension of the analvti« set of zeros of I for ara" ideal 
I of holomorphic function gel'lUS. 
XVe introduce the followilig crucial condition: 
(A) dp(P)  3 for anv point p  Sing(L ). 
This condition is rather weak, in particular generic, as will be explained in Section 
2. 
The following theorems hold in the cat.egories C  and C , with AI a compact 
orientable manifold. 
Theorem 1.1. Let Pt. l  [0, 1], be a famil of Pfaff equations ot ;I . k  2. 
that satisfies the followin9 coditions. 
(a) All P, define the saine characteristic line field L = L(Pt). 
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(b) Ail Pt satisfy condition (A). 
Then there exists a family (Pt of diffeomorphisms of ]il 2k sending Pt to Po. 
Theorem 1.2. Let Po = (Wo) and P1 = (wl) be Pfaff equations on ]il 2¢. k >_ 2, that 
have the same characteristic line field L = L(Po) = L(P). Assume that condition 
(A) holds for Po. Then there exists a diffeomorphism (P sending P to Po provided 
that w is suJficiently close to Wo in the C  topology. 
In the above theorems as well as in Theorems 1.3 and 1.4 below, all objects are 
in the saine category C  or C °, including regularity with respect to the paranmter 
t. The diffeomorphism in Theorem 1.2 can be taken Coe-close to the identity. 
Remark (Closeness of w to w0)). In nmny cases one can present a number r < OE 
(depending on Po) such that the closeness in the C  topology in Theorem 1.2 tan 
be replaced by closeness in the Cr topology. See Theorem 0.1 and Theorem B.2 in 
Appendix B. 
The case n = 2k + 1. The most basic invariant of a Pfaff equation P = (w) on 
/I 2+ is the set 
S = {p e //12k+ : (w A (d)k)(p) = 0}, 
called the Martinet lypersurface, which consists of points at which w is not a contact 
form. This set, invariantly related to P, is the zero level of the function 
/4 =  A (d)/f, 
where ç is a volulne form. 
The ideal (H) of the ring of functions on ]il 2k+, generated by H, is also invari- 
antly related to P. It is called the Martinet ideal. 
The characteristic line field of P = (w) on/12k+ is defined on the set S. 
Definition 1.3. Anv vector field X on/i2+ satisfying the relation 
X]i = o3 A (dw) k-1 A dH mod (H). 
where w is an 3, generator of P, and H is an3" generator of the Martinet ideal, will 
be called a characteristic vector field of P. The line field on S = {H = 0} defined 
by the relation 
p-- L» = {eX(p), a e OE}, p e S. 
is called the characteristic line field of P. It will be denoted by L or L(w) or L(P). 
The set of singular points of L is defined as Sing(L) = {p e M 2+1  H(p) = 
o, x(p) = 0}. 
Above and in the test of the paper the equality of two objects (vector fields, 
differential forms) mod (H) means that their difference is divisible over H in the 
space of objects of the saine category. To check that the line field defined above is 
tangent to S, note that the definition of X implies that XJdH = 0 at any point 
of S = {H = 0} and that X vanishes at any point of S at which the l-form dH 
vanishes. Thus X(p) Ç TpS at any point p  S at which S is a slnooth hypersurface 
and X vanishes at all other points. It is easy to check that the characteristic line field 
is invariantly related to P, i.e., the choices of the generator w of P. the generator 
H of the Martinet ideM, and the volume form Q are irrelevant. 
Note that in the case of the Martinet singularity (p  S and (, A (d,) k- A 
dH)(p) 7(= 0) the definitions of Lp and L coincide with those given in the introduc- 
tion. 
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In order to deal with deeper singularities of P, namely those allowing singular 
points of L, we introduce out invariant dp(P) in the case of odd n as follows. 
Let p E Sing(L). The local characteristic ideal Ip at a point p G /_/2k+l of a 
Pfaff equation P is the ideal Iu(H, X) in the ring of function germs at p generated 
bv the germ Hp of a generator of the Martinet ideal and the coeflïcients 
of a characteristic vector field X of P, in some coordilmte system near p. The ideal 
I u is invariantly related to the germ at p of P = (w) (the choices of a characteristic 
vector field, a generator of the lklm'tinet ideal, and of a local coordinate system are 
irrelevant). The germ at p of tlm set Sig(L) is the zero set of Ip. 
Definition 1.4. Il:  = 2k + 1 and p G S«ug(L), then we define 
du(P) =du(H,-\" ) 
as the maximal length of a regular sequence in the characteristic ideal Ip(H. 
starting with the germ H u as the first element. 
Remarks. (a) In Noetherian rings all nmximal regular sequences in I are of the 
saine finite length, lk[oreover, any regular sequence can be completed to a maximal 
regular sequence. This implies that in the analytic category, independently of the 
parity of n, we have 
du(P) = depth(1 u). 
(b) Similarlv to the case  = 2k, in the analytic category, du(P ) is equal to the codi- 
mension in C of the germ at p of the set of complex zeros of the complexification 
of the idem 
To formulate our reduction theorem for the most general case, we need two prop- 
erties of the Martinet ideal (H): the property of zeros and the extension property. 
Definition 1.5. The Martinet ideal (H) has the property of zeros if for any p 
S = {H :- 0} the ideal in the ring of ail function germs at p generated bv the germ 
Hp of H at p coincides with the ideal in the saine ring consisting of function germs 
vanishing on the germ at p of the set S --- {H -= 0}. 
The property of zeros allows us to identify the hlartinet hypersurface S 
0} with the hla.rtinet ideal. In the case of germs this follows from the definition. 
Examples where the property of zeros is violated at a point p include: 
(a) H u = HîH2. where Hx, H2 are function germs and H(p) = 
(b) H u is equivlent to r 2 = xî +-.-+ x; 
(c) Hp is a fiat germ (i.e., the Taylor series of H at p is zero): 
(d) H u is a zero divisor in the ring of all germs at p. 
In case (c) the property of zeros is violated, since u = r-2Hu is smooth and 
has the same germ of zeros as H u but p ¢_ (Hp). Note that (d) is a particular case 
of (c). 
The local version of the property of zeros (Definition 1.5) implies the global 
version: if a function f on/I vanishes on the set S = {H --- 0, then f belongs to 
the ideal (H). This follows from the fact that division by H or by the germ H u is 
1 
unique (by (d) the germ H u is hot a zero divisor). 

In the Coe-category the global and local versions of the property of zeros are equivalent (this 
follows from the partition of unity). In the real analytic category they are equivalent provided 
that the sheaf of functions vanishing on S = {H = 0} s coherent. In the proof "global implies 
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We also need the e.rtension propert. of the Martinet hypersurface S = {H = 0}. 
Denote by Çoe(l) the Fréchet space of smooth fimctions on AI, equipped with 
the topology of convergence together wfih all derivaiives. Let Ç(,l. S) denoie 
its closed subsl)ace of functions that vanish on S.  define the space of smooth 
funclions on S as the quotienl Déchet space C(S) = COE(kI)/Ç(:II. S). 
Definition 1.6.  sa,y that S has the ea'tensio propert if there exists a COlltillU- 
o,,s linear operator A " Ç(S)  ÇX(kl) such that A(f)[s : f for ail f G COe(S). 
The extension property automat.ically holds in the Ç category, since it holds 
for anv analytic subset S of Al (see [BS] tbr a more general extension theorem). It 
also holds if wo assmne that (H) h, locally aromid any point p G S. a generator 
that is analytic in some coordinate system. 
The fifllowing theorems hold in the categories Coe and Ç, with M compact and 
oriental)le. 
Theorem 1.3. Let Pt.   [0.1], be a famil# of Pfa equatious on M +. k  1. 
that satzsfles the follouring conditions. 
(a) All Pt bave the .saine Mn«'tinet hyper.surface S, which bas the etesion prop- 
erty, ad their Martitct ideals have the propertl of zeros (and cousequetly are the 
(b) All P define the saine chamcteristie line field L = L(Pt). 
(e) Ail Pt satisf9 condition (A). 
The there eists a .family ( qf diffeomorphimns of ./a+ sendin 9 P fo Po. 
Remark. Recall that the extension prol)ery of S holds autonmticallv in the C  
caegory. % conjecture that in the C category the extension property in Theorem 
1.3 also can be omitted. Our proofs show that this is so if le familv Pt h a 
generator  that is polynonfial in t. 
Theorem 1.4. Let Po = (o) ad PI : () be Pfaff equations on M +, h  1. 
u,hieh bave the saine Martinet hypersurface S = S(Po) = ,ç(P) and the saine 
characteristic lie fleld L : L(Po) = L(P). Assune that the Mainet ideal of 
Po bas the property of zeros and Po satisfies condition (A). Then there eists a 
diffeomorphism  sending P fo Po, provided that ' is su«iently close to 'o in 
the C  topology. 
Remarh (Closeness of  to 0)- As in the even-dimensional ce, often one can 
present a mmfl)er r < . (depending on P0) such that the closeness in the C  
topology in Theoreln 1.4 ca. t)e replaced by closeness in the C topology. Sec 
Theorem 0.2 and Theorem B.2 in Appendix B. 
The contents of the rucher .'ections. In Section 2 we explain whv the condition 
(A) is natural and give examples showing that it. cammt be veakened. The conse- 
quences of condit.ion (A) are explained in Section 3 and Appendix A: the condition 
(A) implies certain global division properties of a characteristic vector field. Sec- 
tion 4 contains auxiliarv algebraic statements, which also will be used throughout 
the proofs. Using the division properties and these algebraic staements, ve prove 
Theorems 1.1 and 1.3 in Sect.ions 5 and 7, respectively. The proofs of these the- 
orems are based on the honmtopy method, according to which it suffices to prove 

local" one should use Cartan's Theorem A in [C], which says that any local section of a coherent 
analytic module belongs to the module generated bv global sections. 
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the solvabilitv of the equation 
(HE) Lz, wt q- htcot q- d7 = 0 
with respect t.o a family of vector tiens Zt and a family of fimctions ht (here Lzw 
denotes the Lie derivative of w along Z). Tllen the familv t of diffeomorphisms 
obtained by integrating the familv of vector fields Zt, 
dt 
- Zt(Ot), o = id, 
dt 
transfonns the Pfaff equations (wt) into (w0): 
êt w = tWo, 
where 't = exp(- ]ô t.ds) and t = ht o et- In what follows thc equation (HE) 
will be called the homotopg equatio or bomological equation. 
Theorenls 1.2 and 1.4 are proved in Sections 6 and 8 I)y reduçtion to Theorems 
1.1 and 1.3. In these sections we show that if Pfaff equations Po and P satisfv the 
assumptions of Theorem 1.2 or 1.4. then tlwre exist generators w0 of P0 and  
of P1 suçh that the path of Pfaff equations Pi generated bv wt = w0 + t(w - w0) 
satisfies Ihe assmnptions of Theorcm 1.1 or Theorem 1.3. 
In Al)Iwndix B we present certain topological properties of linear operators re- 
lat.ed to the 5Im't.inet ideal and /he characteristic ideal. W also show a vav of 
transition from the assmnption of C-closeness of w to 0 in Theorems 1.2 and 
1.4 to the C-closeness with a certain r < ,. In the simplest cases this wav leads 
to Theorems 0.1 and 0.2 in the lntroduçtion. 

_9. NECESSITY OF CONDITION (A) 
In this section we explain whv the condition 
(A) dv(P ) >_ 3 
is natural, and we give examples showing that this condition cmmot be weakened: 
if depth dv(P ) = 2. then out theorems are hot truc anvmore. 
Fix a point p E AI a and denote by J the space of/-jets of 1-forlns at p. The 
condition that p is a singular point of the characteristic line field L, i.e. p G Sing(L), 
is the condition 
(u3 /k ((]oE)k--1)(p) = O, 

if n = 2k, and 

(,d A (dw)k)(p) = 0. (,/ (dc6) k-1 A dH)(V) -=- O. 
ifn = 2k+ 1. It involves the /-jet at p of a generator, of P, where i = 1 if 
z is even and i = 2 if n is odd. This condition distinguishes a certain subset 
of J--tlle space of/-jets of w at p. It is hot difficult to see that for any parity 
of n this subset is a stratified submanifold of codimension 3 (see [Mari, [Zh2], or 
[JZh2] for lnore details). Consequently, for generic , the set Siz9(L ) is either 
empty or a submmlifold of AI r of codimension 3. In the real analytic category (and 
conjecturally, in the smooth category too) the set of 1-forlns , violating (A) bas 
infinite codimension in the space of all 1-forms on AIr: see [JZh2], Proposition 3.4 
and Theorem A2 (Appendix 2). 
The following examples show that the conditioll (A) cannot be replaced by the 
condition dp(P) OE 2. In these examples dira M = 4 and diln M = 5. They can be 



2866 B. JAKUBCZYK AND M. ZHITOMIRSKII 

extended to higher dimensions. We have not round an example in the 3-dimensional 
case, but we believe that such an example exists. 
Example 1. Consider the family of Pfaff equations on the 4-torus T 4 generated 
by 1-forlns 
#t = dO + (sin 03 sin 04 + t)d02. 
The characteristic vector field Xt is the sanie for ail t: 
0 0 
Xt = X0 = cos 04 sin 0 3 33 COS 0 3 sin 04 004" 
The set S of singular points of X0 is the union of 8 disjoint 2-dimensional tori (4 
of thcm are described by the equations 0a, 0a E {7r/2, -7r/2}, and the other 4 by the 
equations 0a, 0a E {0,-r}). The codimension of S is 2; therefore depth Ip = 2. The 
restriction of (t) to any of these 2-dilnensiolml tori is a Pfaff equation generated 
by a 1-form ct = dol + ( + t)d02, where 5  {0, 4-1} depending on the torus. 
This Pfaff equation eau be identified with the vector field Vt : (5 + t)O/O01 - 0/002 
defined up to nmltiplication by a nonvanishing function. It follows that the phase 
portrait of I/ on the tortis is invariantly related to (at). It is well known that 
the equivalence of the phase portraits of 1  and t with a fixed 5 implies ri = t2 
provided that t2 is close to tl; see [ArI1] (the parameter t corresponds to the rotation 
number). Therefore the parameter t of the family Pt is a modulus (a parameter 
varying continuously and distinguishing nonequivalent Pfaff equations). 
Example 2. Consider the family of Pfaff equations on the 5-torus 
T(O, 02, ¢1, ¢2, 
generated by 1-forms 
t = (A(0, 02) + Bt(01.02, ¢2)) d¢ + C(01.02)d¢2 + des, 

where 

A(01,02) = 3(sin 01 + sin 02), 
Bi(01,02) = t sin¢2 (1 - cos(0 - 02)), 
C(O. 02) = cos 01 + cos 02. 
A simple calculation gives that t A (dt) 2 = sin(01 - 02)-Qt" fL where f is a volume 
form and Qt is a family of nonvanishing functions on T 5, if t  [-1.1]. Therefore 
the Martinet ideal is the smne for ail t; it is generated by the function sin(01 - 02). 
The Martinet hypersurface consists of two disjoint 4-tori: 
s= îu', î = {o2 =ol}, ' = {o2 =ol + }. 
Since the function Bi vanishes on the torus T14, the restriction of (ut) to T14 does 
hot depend on t. The restriction of (aJ) to the torus T_(¢1, ¢2, ¢3.0) depends on 
t: it is the Pfaff equation (ct), where (ct) = 2t sin ¢2d¢ + d¢3. The characteristic 
vector field of (t) restricted to T is the characteristic vector field of (ct). It is 
2t cos ¢20/001. Assume that t #- 0. Then the characteristic line field does not 
depend on t. The set of its singular points is the union of two disjoint 3-tori T:ï:, 
given by the equations ¢2 = 4-7r/2. The restriction of (at) to T, 3 (or. the saine, the 
restriction of (ct) to T:ï:) is the Pfaff equation of the form (/3t),/3t = 2t5dçbl + d¢3, 
  {-1, 1}. Consider the vector field Vt : -0/0¢1 + 2t0/0¢3 on the 2-torus 
T 2 = T2(¢,¢3). It is easy to sec that the Pfaff equations (,3) and (,3t2) on the 
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3-torus are equivaleut if and only if the phase portraits of t 1 and r 2 on the 2- 
torus are equivalent. As in the previous example, this is so if and only if ri = t2 
provided that tu is close to tl. Therefore the paralneter t of the family (wt) of Pfaff 
equations is a modulus, although these Pfaff equations have the saine Martinet ideal 
(satis[ving the property of zeros) and the saine characteristic line field. The reason 
for that is the violation of the assulnption (A)- the dcpth of the characteristic idem 
is equal to 2 instead of 3. 

3. CONDITION (A) AND DIVISION PROPERTIES 
In this section ve explain implications of condition (A) which will be essential in 
fm'ther proofs. The main implicatious are the following global division properties 
of a characteristic vector field X. As belote, we work in the C  and C  categories. 
Proposition 3.1.a. If a Pfaff equation P = (w) satisfies condition (A). then any 
characteristic vector field X of P bas the following division properties. 
(i) If n is even. then for any vector field " and any r-form u on ]I  with 
r = n-1 or r =  - 2 the equality 
X]u = 0 implies u = 
for an (r + 1)-form p on M', and the eqmlity 
X A t = 0 implies I = fX 
for a function f on M n. 
(ii) If n is odd. then for any vector field }" on 1I ' and any ( - 1)-form u on 
M' the equality 
xJu=o mod(H) imvlies u=xJg mod (H), 
for an n-form # on M , and the equality 
XAI=0 lnod (H) implies Y=fXmod (H) 
for a function f on M'. Here (H) is the Martinet ideal of P, and we assume that 
( H) bas the property of zeros. 
This proposition is a corollary of a general theorem in [DJ] on division properties 
of the interior product with a section X of a vector bundle (see Appendix A for the 
proof). 
We also need a division property with parameters. In the next and all further 
st.atements in this section a 1-parameter falnily of fimctions, differential forms or 
vector fields on M is assumed to be re9ular in t, i.e., depending on t mmlytically 
(in the C  category) or smoothly (in the C  category). 
Proposition 3.1.b. Proposition 3.1.a holds with the forms 
and function f replaced by familles ut, #t, t't, ff, t  [0, 1], provided that in the 
odd-dimensional case either the set S = {H = 0} bas the eztension property (sec 
Section 1) or the familles ut and t't depend on t polynomially. 
This proposition is also proved in Appendix A, using the already lnentioned 
general theorem on division properties. 
Remark. Proposition 3.1 also holds for germs at a fixed point. 
Another implication of condition (A) concerns the structure of the set Sing(L) 
of singular points of the characteristic foliation L: it cannot be too degenerate. 
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Proposition 3.2. If a Pfaff equation (w) satisfles coditio (A) and in the odd- 
dimensioal case the Martinet ideal of (w) bas the propert9 of zeros, then any 
characteristic vector fietd X of(w), the Martinet hypersurface S and the set Sing( L ) 
of singular points o.f the characteristic foliation bave the followin 9 properties. 
(i) If n, = 2k, then the set M \ Si9(L) of points where X does hot vanish (i.e. 
() is quasi-cottact) is dense in M'. 
(il) If  = 2h + 1. then the set iII  \ S (i.e., the set of points af which  is 
contact) is demse in M'. Equivalently, any 9enerator H of the Martinet ideal is 
ot a zero divisor. 
(iii) If  = 2k + 1. the the set S \ Si9(L) is dense in S. 
Pro@ Statelnent (i) follows froln thc observation that if a is liOt quasi-contact 
at ails" point of ali open set, then any charactelistic vector field X vanishes on 
this set. (vanishes on a connected COlnponent of M', in the analvtic category). 
Consequently, givell a point p in this set. the charactcristic ideal Ip at p generated 
by the çoetïficielits of X is trivial and contains no llOn-zero-divisor. This COlitradicts 
assmnl)tion (A). 
Statolnent (ii) is a simt)le inq)lication of the property of zeros of the Martinet 
ideal: see Definition 1.5 raid the exami)les following it. 
To prove (iii), assulne that there exists a neighbourhood U in 3I" of a point p G S 
such tliat a characterislic vector field X vanishes at any point of the set U ç/S. Bv 
the property of zeros of tlie Martillet ideal we obtain that Xp = 0 nlod(Hp), where 
thc subscript indicates the gerln at p. This contradicts assulnption (A) at the point 
p. The proof is COlnplete. [] 
Propositions 3.1 and 3.'2- inlply the possibility of choosing t he Saille characteristic 
vector field for all Pfaff equations with the saine characteristic foliation. 
Proposition 3.3. Let Pt = (--'t), t  [0.1], be a famil9 of Pfaff equations on 
Iii ' satisfyin 9 the assumptions of Theorem 1.1, if » = 2h, or of Theorem 1.3. if 
  = 2k + 1. Then for any family Xt of characteristic vector fields of (wt) we bave 
Xt = RXo, if  = 21,', or 
Xt = R Xo nlod (H), if  = 2k + 1, 
where Rt, t  [0.1], is a family of positive-valued fumtions and Xo i.s Xt with t = O. 
Pro@ Let n = 2],'. Tlie equality L(at) = L(a0) inlplies that (X, A Xo)(p) = 0 for 
ail p G M'. By Proposition 3.1.b we obtain tliat X = RXo, where R is a family 
of functions. Proposition 3.1.b also ilnplies that for any fixed t we liave X0 = QX, 
where Qt is a fimction on _M . This leads to the relation (1 - t?Qt)Xo = O. Bv 
Proposition 3.2. X0 does not vanish on a dense subset of M'; thus RtQ = 1. This 
irai)lies that Rt is a family of nonvanishing fimctions. This falnily is positive value& 
since for any p G/il  the fimction t?t(p) is contiliuous in t and Ro(p) = 1. 
In the case of r = 2k + 1 the equality L(wt) = L(wo) gives (Xt A X0)(p) = 0 
for all p G 5'. From the property of zeros of the Martinet ideal v«e deduce that 
Xt A X0 = 0mod(H). Using Proposition 3.1.b, we see that Xt = t?Xomod(H), 
fol" a fainily of fimctions Rt. Similarly, we have X0 = QtXt mod(H) fol" any fixed t, 
where the Q are filnctions. Therefore, (1 - t?Qt)Xo = 0inod(H). Bv Proposition 
3.2, I?Qt = 1 on S, and so/7 is nolvanishing on 5'. From the fact that Rt(p) is 
continuous in t and ri'oin t?o(p) = 1. we deduce that Re is positive vahled on S, 
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for anv t E [0, 1]. Finally, adding to Rt the flnction ('l-I 2 with a sufficiently large 
constant C, we obtain Rt positive valued on/I. [] 

Condition (A) implies one lnore division property that xve need in our proofs. 
Its proof is postponed to Section 4. 

Proposition 3.4. Let Pt = (wt). t ff [0.1], be o fo,nily of Pfaff eqaatio, on 3I 2k 
satisfying the ass'umt)lio'ns of Theorem 1. l, and let fit be a family of 1-forms such 
that wt A Pt = 0. Then dt = h.twt .for some family ht of fu,wtions. 

Note that this sta.tement is trivial if wt is a falnilv of lonvanishing 1-forlns. but 
we do hot assume this in our theorems. 

4. A('XILIARY ALGEBRA1C LEMMAS 

To prove the solvability of the homot(q)y equation in the pro(frs of TheorellS 1.1 
and 1.3. we will also usc the following siml)lç algei)raic facts. 
Rccall that a 1-form e on 3I ', n _> 3. is called contact (quasi-contact) at p Ç 
M' if n = 2/,: + 1 (resl)ectively, n = 2k) and ((t A (da))(p) # 0 (respectively, 
(c A (d(@-l)(p) ¢ 0). 

Lemma 4.1. Let c aztd A be 1-forms on M 2. ff c is quasi-contact at p and 
(,x / o/ (do) -2) () = ,I. te ( / et)() = O. 

Lelnlna 4.2. Let o be a l-form on/i2k+1. If ( is a contact al p and A is a 1-form 
such that (A A et A (d(Q k-l) (p) = 0 and (A A (da) k) (p) = O. then A(p) = O. 

The facts stated in these lelmlmS are invariant with respect to multiplication of 
a by a nonvmfishilg function, i.e., they are properties of the Pfaff equation (c). 
These properties Call 1)e easily checked in the Dari)oux coordinates in which the 
Pfaff equation takes the forlu (dz + .rldyl +"- + .r,.dy), where r = 
and r=kifn =2/,:+1. 

Lemma 4.3. Let c be a l-.form on ri/2k+l that is hot contact at p, but c(p) ¢ O. 
If A is a 1-form such that (A A a A (do)-)(p) = O. then (A A (da))(p) = O. 

Pro@ We take a nonzero vector v Ç Tpdl 2k+l such that t, Jct = vida = 0. (The 
existence of such a vector follows ri'oto the assumption that o is not contact at p.) 
Then the relation assunled iii the lelmna implies that the form (viA)  c A (dc@ -1 
valfishes at p. It follows that if (o A (da)-)(p) ¢ I). then (vJA)(p) = 0. md 
consequently ( A (da)k)(p) = 0. On the other hand. if (a A (da)t'-1)(p) = 0. then 
the assumption c(p) ¢ 0 implies that (da)(p) = 0, and then again (AA(da)k)(p) = 
0. [] 

Lemma 4.4. /f ft is a volume form on Al ', A is a 1-form, 7 is an (n - 2)-form 
and X is a vector field defined by the relation X J f = A A 7, then X ] A = O. 

Pro@ To prove this statelnent, note that the definition of X ilnplies X J (AAT) = 0, 
and consequently (X ]A) -7-t- (X ]7) AA =0. It follows that (X ] )" (AAT)=0. 
Since X valfishes exactly at points at which the form A A7 vanishes, we obtain that 
xii=o. [] 

Finally, we need the following general properties of a characteristic vector field. 
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Lemma 4.5. If X zs a characteristic vector field of a Pfaff equatzon (w) on 1I 2k, 
tl en 
X]w = 0 and (X]d,z) Aw = 0. 
Pro@ The first relation follows ri'oto the definition of X and Lemma 4.4. The 
definition of X implies that X] (w A (dw) '-) = 0, which, together with X]' = 0, 
gives (XJdw) A w A (dw) k- = 0 if k > 1. Now the second relation in Lemma 4.5 
follows froln Lemma 4.1 at points where w is quasi-contact. At all other points the 
field X vanishes, and thcre is nothiug to prove.  
Lemlna 4.6. [f X is a characteristic vector field of a Pfaff eqatio (w) on M 2+1 
wlose Martinet ideal bas the propertg of zeros, and H is a generator of this ideal. 
then 
XJw=0 mod (H), XJdH=O mod (H), (XJdw) Aw=0 mod (H). 
Pro@ Due to the I)roperty of zeros of (H), it suffices to prove the three relations 
at any point p of the Martinet hypersurface S such that X(p)  O. The relation 
(XJdH)(p) = 0 follows immediately from the definition of X. To see the other two 
relations, note that S is regular in a neighbourhood of a point p such that X(p)  O. 
The definition of the characteristic vector fiel(1 X in the case n = 2k+ 1 implies that 
the vector field X]s on S is, in a neighbom'hood of such a point p, a characteristic 
vector field of the Pfaff equation (w]s) on S (which is qui-contact at p). Thus 
the remaining two relations follow from Lemma 4.5. The proof is complete.  
Proof of Proposition 3.4. Let Xt be the characteristic vect.or field of (wt) defined 
bv XtJ = wt A (dwt) -. Since Xt (and so wt) does not vanish on a dense sub- 
set of al , the condition t A/t = 0 and Lemma 4.5 imply that Xt]flt = 0 and 
Xt] (fit A (dwt) k-I) = 0. From Proposition 3.3 we have the equality Xt = RtXo, 
with Rt nonvanishing; thus X0J (fit A (dwt) k-l) = 0. Therefore the division prop- 
erty in Proposition 3.1.b implies the following relation: fit A (dt) k- = No ] Pt = 
(gt/Rt)Xt ] , where  is a volume form, gt is a family of flmctions and t = gt. 
Taking ht = gt/R, we tan rewrite this relation iii the form 
(t - brut) A (dt) - = O. 
Let us show that this relation implies fit- tt = O.  know that (fit- htwt)Awt = 
0, since fit A wt = 0. Fix t and a point p at which wt is qui-contact. At this point 
wt does hot vanish: therefore (fit - btwt)(p) = rwt(p), with the scalar r depending 
on t and p. Then the displayed relation implies that r(wt A (dwt)k-)(p) = 0 and 
consequently r = 0. So, (fit - htwt)(p) = 0 if p is a point at which wt is quasi- 
contact. By Proposition 3.2. (i) the set of such points is dense, and so fit = htt at 
any point of the manifold. The proof is complete.  
Now we are ready to prove the solvability of the homotopy equation (HE) and 
out main theorems. 

5. PROOF OF THEOREM 1.1 
Solvability of the homotopy equation (HE) in Section 1 is equivalent to solvability 
of 
( dwt) )k-2 
(5.1) Lz, a.'t ÷ -- Awt A (dw't =0, 
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with respect to a fmnily Zt of vector fields. NaInely, equation (5.1) is obtained from 
the homotopy equaltion by extlernal multiplication by wt A (dt) -2. Cnversely, if 
(5.1) is solvable then, using the fact lhal the set of quasi-contacl points of (wt) is 
dense in BI" (Proposition 3.2.(i)), we get from (5.1) by Lelnma 4.1 that (Lztwt + 
(dwt/dl))Awt = 0. Therefore, by Proposition 3.4 we get Lz, wt +(dt/dt)+htt = 0. 
for a fmnily of fllnctions ht, which is the holnotopy equat.ion (HE). 
A solution Z, of equation (5.1) will be constructed within the set of fmnilies Zt 
satisfying 
(.2) z, ] , = 0. 
Condition (5.2) ilnplies that Lz, wt = Zt ] dwt, and the equation (5.1) tan be 
rewritten in the forln 
dt )k- 
(5.3) Z](wtA(dt)-l)+('-l)AwtA(d, =0. 
In order to solve equation (5.3} we fix a vohlnm form tl and define a familv Xt of 
characterist.ic vector fields of (wt) by the relaion Xt ] Q = wt A (dwt) k-l. Lemlna 
4.5 and Proposition 3.3 imply the relations XoJwt = 0, XoJ(dwt/dt) = 0 and 
(Xo]dwt) A wt = 0. Thus 
X0]ut = 0. where oet =  Awt A (dt) - 
Therefore, by the division property in Proposition 3.1.b, we ham 
with solne Nlnily fit of (n - 1)-forms of the saine regularity with respect to t as in 
wt. Using Proposition a.a again, we obtain 
for some, regular in t, family of ( -- l)-forms pe. This relation allows to rewrite 
equation (5.3) in the form 
&](X,] )+(-I)X,],=0. 
The latter equation bas a solution Z« defined by the relation 
It now renmins to check that the construçted solution Z satises relation (5.2). 
The equality (5.2) is equivalent to the relation  A   0. From (5.5) and the 
definition of oe we bave (X ] ) A   0. By Lemma 4.5. XJwt  0; therefore 
X ] ( A )  0. Thus the n-form  A « vanishes at any point at which X« 
does not vanish. By Proposition 3.2, (i) the set of such points is evervwhere dense; 
therefore p« A   0 and (5.2) holds. This completes the proof of Theorem l.l. 

6. PROOF OF THEOREM 1.2 
We will use the following proposition (its proof is postponed to the end of this 
section). 
Proposition 6.1. Assume that Po = (w0) satisfles condition (A) and 
(6.1) w/ (d) k-1 = w0/ (da0) k-l. 
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Tbe for the patb  = (1 - t)wo + t we bave 
(6.2) cor A (dct) k-1 = A t 0 A (d0) k-l, 
where Af is a family of functions, polynomial i t. 
Proof of Theorem 1.2. The equality L(0) = L() implies X0 A X = 0. where 
Xo and X are characteristic vector fidds of P0 and P1- Condition (A) satisfied 
for (0) enables us to use the second division property in Proposition 3.1.a, (i) to 
dedu«e that X = RXo and. equivalently, 
(6.3)  A (dWl) k-1 = R0 A (dw0) k-l, 
where R is a Slnooth or analvtic flmction. In fact, R is positive value& which wil] 
follow ffoto the closeness of wa to w0- Therefore. assunfing R > 0, we choose the 

gellerator 

and we have 
(6.4) 
Let 
(6.5) 

£1 / (d&l) k-1 = '0 / (dw'o) k-l- 

t = (1 - t)co0 +/I- 
To prove Theolen 1.2 it is sufficient to show that the family of Pfa.ff equations 
satisfies the assumptiols of Theorem 1.1. 
The equality (6.4) allows us to use Proposition 6.1 to conclude that the relation 
(6.2) holds for the path (6.5). It is «lear that (6.2) implies that the familv (wt) 
satisfies the assmnptions (a) and (b) of Theorem 1.1 provided that the fun«tions 
At in (6.2), t G [0, 1], vanish at no point of 3I . This will follow froln the assump- 
tion on the C-closeness of wt t.o 0 and Theoreln B1 in Appendix B. Define a 
characteristic vector field Xt of (wt) bv the relation XtJQ = t A (&t) -1. where 
Q is a vohune forln, By (6.2) we bave Xt = AtX0. The C-closeness of '1 O '0 
implies the C-closeness of Xt, t Ç [0, 1], to X0. Bv Theol'eln B1 the C-closelmSS 
of Xt to X0 in the equality Xt = AtXo ilnplies that the function At, t  [0.1], is 
C-«lose to 1. ÇOlsequently, Af valfishes at no point of the lnanifold. The proof 
of Theorem 1.2 is COml)lete. 
Proof of Proposition 6.1. Using (6.1), we may assume that the chara«teristi« vector 
fields Xo and X of (#0) and (#), respectively, are equal.  shall prove that 
(6.6) X0 J (w, A (dwt) k-l) =0. 
Having (6.6), we can use assumpt.iox (A) and the division property in Proposition 
3.1.b (wit h polynolnial dependence in t), which gives wt A (dwt) k-  = XoJ Pt, where 
pi is a volume form. Let 'o A (d#o)  = Xo].  = A. Then we get (6.2). 
To pi'ove (6.6), we note that bv Lelmna 4.5 we have 
(.7) XoJ,o = xOJ«l = 0; 
therefore XoJwt = 0. It follows that in order to prove (6.6) it suffices to prove the 
equality 
It is enough to prove the equality (6.8) at any point p such that Xo(p) ¢ O. At 
such a point o(P)  0 and, since Xo = X1, (p)  O. From Lelnlna 4.5 we have 
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(XoJdwi) A wi = 0, i = 0, 1. Thus, there are flulctions h0 and hl, defined in a 
lieighboul'hoo«l of p, sUC|l that iii this imighbourhood ve bave 
(6.9) XoJdwo = howo. XoJdw = hll. 
XX will prove below that 
(6.10) h t = h. 
Thell frolll (6.9) we get XoJdwt = hwt, where h = h0 = bi, and so (6.8) holds. 
We will show that (6.10) follows ffoto (6.1}.  take thc Lie derivative of both 
parts in (6.1) along the veçtor field X0. Using thc forlnula Lxq = d(XJ q) + XJdq 
for the Lie derivative, we obtain 
= 0 + k(X0Jdw0) A (da0) -1 = t'haa0 A (dwo) -1, 
and siinilally 
L.,- (1  (dl) -') = t,,l  (1) *-1. 
Conq)al'ing these equalities and using (6.1) again, we gel the required relation (6.10} 
(since w0 A (dwo) t- = w A (dw) t-I does llOt vanish on a dense subsçt of 3I). 
Proposition 6.1 is proved.  

7. PIROOF OF T.HEOREM 1.3 
Since the lklartiiml hypersurfaces of Pt = (wt) are the samc for all t. the lklartinet 
ideals are the Saille bv t.]le I)roI)erty of zeros. Thus we Call fiX a generator H of these 
ideals. The following two propositions will hold ulder the assumptions of Theorenl 
1.3. In the propositions ail fanlilies are regular with respect to t (slnooth in the 
C  category and almlytic iii the (7" category). 
Proposition 7.1. There ezists a familg of vector fields l't satisfging the relation 
( 
(7.1) Ly, w, + dt J Awt A (dw,) k-1 =0 mod (H). 
Proposition 7.2. Let pt be a familg of 1-forrns such that 
(7.2) tzt A Wt A (dcdt) k-1 = 0 niod (H). 
Then the equation 
(7.3) Lztwt 
bas a solution ( Zt, ht ) . 
The solvability of the honiotopy equatioli (HE) in Section 1 is a direct corollary 
of these propositious. Nalnely, we take ttt = --Lytcd t -- dwt/dt, and then the pair 
(2t, ht), with 2t = Zt + t, solves the holnotopy equation (HE) 
Proof of Proposition 7.1. We fix a vohlme form t and define a falnily Xt of char- 
acteristic vector fields of (wt) by the relation XtJf = wt A (dwt) k-1 A dH. Frolll 
Proposition 3.3 we have 
X t = tt.\" 0 lllod (H), 
where/t is a familv of nonvmiishing fllllctiollS, regular in t (of the saille regularity 
in t as in the falnitv wt). Bv Lenima 4.6 we bave XtJwt = 0 mod (H) and 
{XtJdwt) A wt = 0111od (H). V'P Illay repla.ce X with Xo iii these equalities, hl 
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particular, we get X0]wt = 01nod (/-/), which ilnplies Xo](dwt/dt) = [Imod (H). 
Taking ail these equalities into account, we see that 
(7.4) X0]  Awt A(dwt) -1 = 0 mod (H). 
This equality and Proposition 3.1.b inlply that 
dwt 
d A wt A (dwt) k-1 = Xo J (f,) mod (H), 
where ft is a falnily of fimctions, regular in t. Replacing X0 with RÇ1,Yt and using 
the definition of Xt, we see that we can rewrite this relation in the forln 
 A wt A (dwt) k-1 
dt 
where gt = ft/Rt. This allows us to rewrite equation (7.1) in the form 
(7.5) (L,wt - gtdH) A wt A (dwt) -1 = 0 mod (H). 
It is clear that (7.5) hohls if  satisfies thc relations 
(7.6) l) J dwt = O, kt J wt = gtH. 
since in this case 
Lwt = d(})J w) = d(gtH) = gtdH mod (H). 
Since (H) is the Maltinet idem of (wt), we have 
(7.7) 
for a falnily St of nonvanishing functions which has the sanie regularity in t  in 
wt (this follows froln the regularity of the leh-hmld side and the fact that division 
by H is a continuous linear operator in the space of Slnooth fnllctions, see Theorem 
B1 in Appendix B). Let us show that (7.6) holds for the family }) defined by 
(7.s)  ] = (d) . 
In fact, applying }] to (7.8), we get gt(}Jdt)A(dwt) - = 0. This relation ilnpli 
 J dwt = 0 (at points where g,(p) = 0 we have }(p) = 0, and at other points we 
can use Lelnnla 4.2 with A = (]dt) and the fact that contact points are dense). 
We have shown the first equality in (7.6). hl order to prove the second one we 
apply })] to (7.7) and, using (7.8). we obtain that (}] wt- gtH)" (dwt)  = 0. This 
implies that ) ] wt - gtH = 0 at points where the form (dt) k does not vanish, in 
particular, at points where wt is contact. By Proposition 3.2, (ii) the set of such 
points is everywhere dense; therefore }]wt -gtH = 0 everywhere, and so (7.6) 
holds. Proposition 7.1 is proved. 
Proof of Proposition 7.2. Bv Lemma 4.2, (i) and the fact that the set of contact 
points is dense in BI  (Proposition 3.2. (ii)), the equation (7.3) reduces to the 
following tvo equations: 
(7.9) (z,) ,  (d,) - = ,, '  (d.,) «, 

(7.10) (Lz, w, + h,,) / (d,,) k = ,, / (d,,)  
(with unknown Zt and ht), obtained from (7.3) by external multiplication by the 
forlns wt/ (dw,) -1 and (dwt) , respectively. 
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To solve equation (7.9) we use assmnption (7.2). By this assumption 
(7.11) Pt A cor A (dwt) k- = Hut 
for some familv u of 2k-forms, regular in t })y Theorem B1 in Appendix B. This 
permits us to find an explicit solution Z of (7.9). Namely, since H is a generator of 
the Martinet idem of (w), we bave relation (7.7), i.e., w A (dw)  = H&[, where 
& is a fmnily of nonvanishing flmctions, regular in t. Let ris show that the family 
of vector fields Z defined by the relation 
k 
is a solution o[ thc equation (7.9). e]at.ion (7.1]) and the fact that is hot a zem 
divisor imply that ut A wt = 0. This and (7.12) imply that 
z, ] , = (. 
Consequently, Lz, wt = Zt ] dwt and 
(r.la) (Lz, co,)co,(d,) -' = (Z, J d,),(a,) -1 = Z, J (#,  (d,/). 
k 
Now (7.9) follows from equalities (7.7) and (7.11)-(7.13). 
To prove Proposition 7.2 it renmins to solve equation (7.10) with respect to 
Since Zt]wt = 0. then (Lz,t) A (dt)  = (k + 1)-Zt](dt) TM = 0, and the 
equation (7.10) takes the form 
htwt  (d,)  = vt  (d#t) . 
Due to relation (7.7), to prove that this equation has a solution ht it suces to 
prove that #t A (dwt)  = HCt, where Ci is a family of fimctions, regular in t. 
shM1 first prove that 
(r.14) (,(d,) )(v)=0. f VeS- 
This follows from relation (7.2). Namely, since t is hot contact at a point p  S. 
thus (7.14) follows from (7.2) bv Lenmm 4.3. provided that wt(p) ¢ 0. Since the 
set of points of S at which wt vanishes is a subset of the set Sig(L), the set of 
points p  S where wt(P) ¢ 0 is dense in S by Proposition 3.2. (iii). Therefore 
(7.14) holds at ail points p  S. By the property of zeros of the ideal (H) we obtain 
t A (&t)  = HCtiL Since t and t are regular in t, we deduce from Theorem B1 
in Appendix B that Ct is regular in t. Proposition 7.2 is proved. 

We have completed the proof of Theorem 1.3. Note that the extension property 
of S was used only when referring to Proposition 3.1.b., and therefore it is hot 
needed if cor is polynomial in t (cf. the remark after Theorem 1.3). 

8. PROOF OF THEOREM 1.4 

Since the Martinet hypersurfaces for Po = (oz0) and P = (COl) are the same and 
the Martinet ideals have the property of zeros, they are equal and we tan choose 
a common generator H which will be used throughout the proofs. The following 
proposition holds under the assumptions of Theorem 1.4 and will enable us to 
reduce the problem to Theorem 1.3. 
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Proposition 8.1. Assume that 
OE1 / (doE1) k-1 / dH = wo A (dwo) k-1 A dH 
Ttem for the potl t = (1 - t)wo + hZl we Iave 
Wt A (dwt) k-1 A dH = Bt wo A (dwo) k-1 A dH 

(8.1) 
(8.2) 

mod (H) 

mod (H), 

SO 1 A(doel) k-1 A dH = co0 A (dw0) k-1 A dH mod (H). 

cor = (1 -- t ) wO q-tdOl. 
To prove Theoreln 1.4 it is enough to show that the family of Pfaff equations (wt) 
satisfies the assuml)tions of Theorem 1.3. Note that we do hot need the extension 
property of ,5'. since the familv cor in (8.6) is polynomial (in fact. affine) in t, and 
in this case Theorem 1.3 was I)roved without using this assmnption. 
The equality (8.5) enables us to use Proposition 8.1. h is clear that relations 
(8.1) and (8.2) imply that the falnily (wt) satisfies the assmnptions (a), (b) and 
(c) of Theorem 1.3 provided that the fimctions Bt do hot vanish on S (then the 
characteristic line field does hot change) and Ct do hot vanish on 3I ' (then the 
Martinet ideal does hot change), for t  [0, 1]. 
The fact that Bt and Ct do hot vanish follows from tlle 
0 and Theorem B1 in Appendix B. Since CO 1 is C-close 
field Xt is C close to X0. The relation X = /?X0 mod 
the inverse to the operator Lx.H in Theorem B1 imply that 
/) that is C-close to 1 and equal to  at anv point of the 
S. Bv the property of zeros of the Martinet ideM, /) = 
replace  bv / in (8.3) and in the definition (8.4) of &l- 
CC-close to CO0- Define a falnily of characteristic vector fields Xt bv the relation 
XtJçt -= COr A (dCOt) k-1 / dH. Then Xt,t  [0, 1], is Coe-close to X0. The equality 
(8.1) is equivalent to Xt = BtXo mod (H). We again use continuitv of the inverse 
to the operator Lx.H in Theorem B1. By this theoreln there exists a fimction /t 
that is C-close to 1 and equal to Bt at any point of S. Therefore Bt > 0 at any 
point of S. 

(8.4) 
and we gct 
(.5) 
Let 
(S.) 

COr A (dcot) k --- Ct CO0 A (dCO0) k, 
wher'e Bt ad Ct are fizmilies of flnctions, polynomial in t. 
Proof of Thcorem 1.4. Let X0 and .k' be characteristic vector fields of P0 and 
P defined via the saine volume tortu and thc saine generator H of the Martinet 
ideal. Since L(wl) = L(wo), then X A X0 = 0mod (H). Froln condition (A) 
and thc division property in Proposition 3.1.a we obtain X = HX mod (H) or, 
equivalent ly, 
(8.3) 1  (dwl) k-1  dH = RWo A (dw0) k-1 A dH mod (H), 
where  is a smooth or analvtic flmction.  will later show. using closeness of w 
to w0. that  is positive vahmd. Thcrc%re, we can change the generator of Pt for 
1 = R/w, 
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To prove that Ct vanishes at no points of àl n, we also lise the C°°-closeness of 
cet. t E [0, 1], to w0 showll above. Let wt A (dwt) k = QtHf, where  is a volume 
form. Relation (8.2) iml)lies that Qt = CtQo. The C°-closeness of cet to co0 implies 
the C-closeness of the fimction HQt to HQo. By continuity of the inverse to the 
operator f ---, fil (Theorem BI) we get the C-closeness of Qt to Qo. Since Q0 
is a nonvanishing flmction, tlwn Ct is Coe-close to 1. The proof of Theorem 1.4 is 
complete. [] 

Proof of Propositio 8.1. It is enough to prove the equalities 

-\'o J (cetA(dce) k-1AdH)-----Il mod (/4), 

Namely, eqnality (817) and conditi»n (A) allow us to use the divisim properties in 
Proposition 3.1.b to conchlde that cor A (dcot)/'-1 A dH = Bt.\']t2mod (Il), and 
so the validitv of relation (8. l), where Bi is a famih of fimctions, polynomial in 
t. Equality (8.8) ilnplies that cor A (dcot)  vanishes at those points of S at which 
X0 does hot vanish. By Proposition 3.2, (iii) the set of snçh poiuts is dense in S: 
therefore cor A (dcet)  vanishes at all points of S. By the property of zeros of the 
Martinet ideal we have cet A (dcet)  = 0 mod (H), and consequently (8.2) holds for 
SOlne familv of fnnctions Ct. polynomial in t. 
In order to prove (8.7) and (8.8) we use the assmnption of Proposition 8.1 and 
choose characteristic vector ficlds X0 and XI of (co0) and (coi) eqnal modulo (H). 
By Lelmna 4.6 we have 

.\]w, = 0 mod (H). XoJdH = 0 mod (H). 

for i = 0, 1, and therefore X0Jwt = 0 mod (H). It follows that in order to prove 
(8.7) and (8.8) it suffices to prove the equality 

(819) (Xo]dcet) A cet : 0 mod (H). 

Due to the property of zeros of the Martinet ideal, it suffices to prove this equality 
for any point p Ç S such that Xo(p) 7  O. At snch points the 1-fonn dH does hot 
vanish and S is smooth. Since XI = -\-o mod (H), using Lemma 4.6 we obtain 

(Xo]dceo) A o = (X0Jdce) A co I : } 

in a neighbom-hood of p in S I Since X0(p) ¢ 0. then oto(p) -¢ 0 and cel(P) ¢ 0, and 
therefore these relations imply the equalities 

(8l 10) 

XoJdce0 = hr.cco, XoJdvl --- ]licol, 

which hold iii a neighbourhood U of p in S. Here ]10 and ]il are functions defined in 
this neighbourhood. We will show that ho = hl; then (8.10) implies that Xo]dcet = 
]10cot, and (8.9) holds in the neighbourhood U. 
To prove that ho = ]il Oll []" C S, we restrict the relation assmned in Proposition 
8.1 to the tangent bundle of . We obtain (coi A dcel)[u = (co0 A (dco0)k-)l and 
take the Lie derivative of this relation along the restriction Xols of X0 to S (recall 
that X0 is tangent to S). As iii the proof of Proposition 6.1, we obtain the required 
equality ]10 = bi. Proposition 8.1 is proved. [] 
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APPENDIX N. DIVISION PROPERTIES 
In this Appendix we present a general theorem on division properties of the 
exterior (respectively, interior) product with a section X of a vector bundle. This 
theorem is proved in [DJ] and implies our Propositions 3.1.a and 3.1.b. Our results 
hold in the categories C s, where s = ex2or s = w. 
Let AI be a paracompact differential manifold. Consider a vector bundle E over 
'I of tank m and denote by E* its dual bmdle. Let Af = Af(E) denote the rth 
exterior power of E, r = 0, 1 ..... m, with A0 =/I x  and Ai = E. We denote , 
A(M: E) the linear space of sections of A,, (smooth or real analytic, depending on 
the category). 
Any section cv of E defines lhe linear operator of exterior multiplication bv c. 
which gives the complex 
(A.I) 0 --, A0(M) ---, A(M: E) --, ...... --, A,,(M: E), 
wflh the operator 0« = 0 : Av(M: E) --, Av+(M: E) defined by 0(')') := o A % 
Consider a section X of the dual bundle E*. This section defines the operator 
of the interior product with A', Ni : A,.(M: E) --, A_I(M:E). Given a local 
basis e .... , e of E, the operator of the interior product with X is defined on the 
elements of a local basis of A bv 
xJ(%  ...  e) = (-1)-' (x. %) «,  ...% ...  e, 
3=1 
where êi means absence of ei mld (-, .) denotes the dualitv product between E* 
and E. Clearly, (X])2 = 0; so the operator A'] defines the complex 
(A.2) 0--, A,(AI: oe)--, A,_a(AI: oe)--, ...- Al(AI:E)-* Ao(M: E). 
Let S be a closed subset of .àl. Denote by A(M. S: E) C A,.(M: E) the subspace 
of sections of Af(M: £) vanishing at ail points of S. and let 
A(S: E)= A,.(AI: E)/A,.(AI, S; E) 
denote the quotielil space. 
Any element  of AI (ç; E) defines the unique operator 
0 : Ap(S: E) --, Ap+a (S: E) 
(the quotient of the operator of exterior multiplication), which gives the complex 
(A.3) 0--, Ao(S)--, A,(S: E)--, ...... --, A,(S: E). 
Given a section X of £*, the operator XJ defines the following complex on the 
quotient spaces: 
(A.4) 0--- Ara(S; E) Am-I(S; E)---.--- A(S: E)-- A0(S: E). 
We define the invariant dp(X) = depth(Ip), where Ip is the idem of function 
germs at p G M generated by the coelïïcients a ..... a of X in a local basis of E* 
(cf. Definition 1.2 in Section 1). Sinfilarly, given a pair (H, X) of a flnction H and 
a section X of E* on M, we define dp(H. X) as the maximal length of a regular 
sequence of function germs that begins with the germ Hp of H at p and has further 
elements in In (cf. Definition 1.4). Analogously we define the invariants dp(a) and 
dp(H, 
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Statements (i) and (il) of the following theorem hold in the C  and C  cate- 
gories, for 0 _< q _< n- 1. 
Theorem A. (i) If « satisfles the condition dp(o) >_ q + 1 for all p E .I such 
that c(p) = O. thon the complex (.4.1) is exact up fo Aq(M:E). Similarlg. if 
dp(X) >_ q + 1 for all p  ]il such that X(p) = O. then the complex (.4.2) is exact 
up to Am_q(M; E). 
(il) Let H be a function on ]il such that lhe ideal (H) bas the prop«rty of zeros. 
and let S = {H = f)}. If et is a section of E on M such that (H.() satisfies 
d(H. c) >_ q + 2 for all p  S such that et(p) = O. thon the conplex (.4.3) is exact 
up to Aq(S; E). Similarl,q. if X is a section of E* on M and d(H, X) >_ q + 2 for 
all p  S such that X(p) = O. then the cornplex (.4.4) is e.ract q fo A,_q(,ç; E). 
(iii) If the assumptions of (i) hold, then. in the C  category, the cornplex (A. 1) 

splits up to Aq-l(]il;/) and 
assumptions of (il) hold and 
splits up fo Aq-l(S;/) and 

t/Le complex (.4.2) split. up to Am-q+] (]il;/). If the 
S bas the extension propertg, then the conpleï: (A.3) 
the complex (A.) splits up to Am-q+l(S; E). Here 

the correspoding spaces are equipped with the C  topology and are considered as 
Fréchet spa«es (quotient FfCher spaces). 
Above, a complex 0 -- L, -- .-. -- Lrn-q+l -- Lm-q -- ..- defined by the 
operators Oi : Li -- Li-1 is called exact up to Lm-q if Im0i+l -= ker Oi for i = 
m, m-- 1 ..... m --q, and it splits up to Lin-q+1 if the L are linear topological spaces. 
hnOi are closed subspaces of Li-1 and each Oi : Li -- Li-1 has a continuous right 
inverse tçi defined on hn Oi, for all i = m, m - 1 ..... m - q + 1. 
The above theorem follows rioto The(»rems 2.1 and 2.2 in [DJ]. In the local 
case (of germs) statements (i) and (il) follow from a well-known algebraic result on 
exactness of the Koszul complex, cf. e.g. lE] or [JZh2], A1)pendix 1. 
Proof of Proposition 3.1.a. In the even-dimensional case the first implication fol- 
lows trivially from statement (i) in Theorem A if we take the bundle E equal to the 
cotangent bundle E = T'M, the dual E* = TM, and consider the complex (A.2). 
(A(M, E) is identified with the space of diffcrential r-forms on ]il.) The second 
implication follows analogously from the saine statement by taking E = TM and 
the complex (A.1). 
In the odd-dimensional case the first implication follows in a similar way from 
statement (il) in Theorem A concerning the complex (A.4). This is because the 
property of zeros of (H) allows us to identify the elements of A(S: E) with the 
equivalence classes of differential r-forms modulo (H) (cf. out convention on no- 
tation mod (H) presented after Definition 1.3). The second implication follows 
analogously from statement (il) in Theorem A concerning the complex (A.3). [] 
Proof of Proposition 3.1.b. The existence of pt and ft for any fixed t follows from 
Proposition 3.1.a. We have to show the regularity of these familles in t. We shall 
prove the regularity of ttt (the proof of regularity of ft is analogous). If ut depends 
on t polynomially, then Proposition 3.1 allows us to construct #t polynomial in t, 
and the regularity follows trivially. In the general case out argmnents are different 
for the categories C" and C . 
In t.he C" category we use the following fact: if a sequence al ..... a, of rem 
analytic function germs at p  ]il is regular in the ring of analytic function germs 
at p, then it is regular when considered as a sequence in the ring of real analytic 
function germs of the variables (x, t)  M × 11¢ at (p, to), for anv t0 6 [0, 1]. Using 
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local coordinates, this fact can be easily proved for the case of formal power series 
using the definition of regular sequcnce in the ring of fornml power series of the 
variables x ..... xn, t. Then, using the fact that the ring of formal power series 
is faithfully fiat. over the ring of convergent series (see Malgrange [Mlg], Chapter 
3), we see that it also holds for converging series and so for germs of analytic 
functions. Using the above fact we see that the assumption (A) holds over the 
manifold ,ÇI = M x I, where I is an open interval containing [0, 1] on which the 
analytic family vt is well defined by analytic extension. Thus we Call lise Theorem 
A over the manifold .). i.e., for the bundles E = T*M and E* = T3I pulled back 
to IÇI by the canonical projection M x I  21I. 
In the (' category, in the even-dimelsional case the smooth dependence of 
pt on t follows rioto statement (iii) in Theorem A. By this statement there exists 
a continuons right inverse operator K to the linear operator X] : A,.+(M: E)  
A,.(M: E), for r= n-1 and r = n-2, and we can define pt = Kut. Here E = T*M 
and A,.(M: E) = A(M). the space of diffcrential r-tbrms on M. 
In the Coe category, iii the odd-dilnensional case we also use statement (iii) of 
Theorem A and the extension property of S. Namely, fbr E = T*3I we define 
lt = A Kut]s, where Iç : A,-I(S; E)  A(S: E) is the continuous right inverse 
p,'t,r to xJ : .,,(s; )  &,_(s; ). ,,d  : .%(5'; )  .,,(M: ) is  
contimums linear operator of extension.  

APPENDIX B. CONTINUITY OF DIVISION 
Contilmity of division in the cases presented below is needed in the main proofs 
and will be proved separately. Let C(.I) and l'ect'(M) be the spaces of smooth 
functions and smooth vector fields on M. with the C topology. Let C(3I, S) 
and lect'(M..ç) be the subspaces of flmctions (vector fields) on 3I vanishing on 
the Martinet hypersurface S c M. The quotient Fréchet spaces 
C(.ç)=C(M)/C(M..ç), Vect(S:TM)= Vect(M)/Vect(M.S) 
can be idcntified with the space of smooth function on S and the space of blnooth 
sections of the tangent bundle T3I restricted to S. rcspectively. 
Given a Pfaff equation on 3I 2 and a characteritic vector ficld X, we consider 
the linear operator 
Lx : C(M) -+ lect(M). Lx(f) = fX. 
For a Pfaff equation on M t+. a characteristic vector field X and a generator H 
of the Martinet idem we consider the linear operators 
/: C(M)  C(M)  ;(I) = 

and 

where [  ] denotes the equivalence class in the corresponding quotient space. 
Theorem B1. If the characteristic vector field X satisfies codit.io (A ) and in the 
odd-dimesional case the Marrinet ideal (H) has the propert9 of zeros, then each 
of the lmear operotors Lx. Lt4 a, Lx.tî, is bjective, bas closed ima9e, and hos 
codinuous im,erse defined o the image. 
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Pro@ By the Banach open lnapping theorem in Fréchet spaces it suffices to prove 
that each of the operators L,\- LH, and Lx. is injective and has closed image. 
The injectivity of the operators Lx, LH a.nd L.\-. follows from Proposition 3.2. 
(i), (il), (iii), respectively. The closedness of the image of the operator L follows 
h'om the global property of zeros inlplied bv Definition 1.5 by this property the 
image of LH coincides with the closed subspace C(M, S) c C(M) of hmctions 
vanishing on S. 
To prove the closedness of the image of the operators Lx and L,\-., we use 
Proposition 3.1. By Proposition 3.1. (i) the inmge of Lx coincides with the kernel 
of the continuous operator Vect(M) --+ V-'(A"-TM) defined bv  -- X A]t, where 
F(A2TM) is the space of smooth scctions of the skew-symmetric product of the 
tangent bundle TM, with the C  topology. Similarly, hv Proposition 3.1, (il) the 
inmge of Lx.n coincidcs with the kernel of the continuous operator l'ect'(S) -- 
F(A2(S:TM)) given by [Y] --+ IX A }'], where F(A"-(S:T3I)) is the space of 
smooth sections over ,_q of the skew-swmnctric pro(hlct of the t.angcnt bundle T3I 
(with the C  topotogy) and [  ] denotes the equivalence çlass in the corresponding 
quotient space. Thc kernet of this op('rator is a closod subspace of l'cet ' (S: TIII) = 
Vectoe(l)/l'ectoe(M, S). The proof is complete. [] 

It is natural to ask if it is possible to replace the Coe-closeness of ,Zl to w0 in 
Theorems 1.2 and 1.4 by C-closeness with some r. Any attempt ai answering 
this question requires nlodification of Theorem B1, which was used in the proofs 
of Theorems 1.2 and 1.4. Proving Theorem 1.2. we had to show that the function 
AI. t  [0, 1], does hot vanish at any point of M. In the proof of Theorem 1.4 we 
had to show that the functions BI, I  [0, 1], do hot vanish ai points of S and the 
hmctions Ct, t  [0.1], do hot vanish ai points of 31. The C-ctoseness of 1 to d 0 
given as an sumption in Theorems 1.2 and 1.4 and the continuity of the inverse 
to the operators Lx, Ln and Lx.n allowed us to obtain the C-closeness of AI, 
BI and C, to 1. Of course, to «onclude that these hmctions do hot vanish, their 
C°-closeness to 1 woutd be enough. 
ç introduce the following topological chara«teristic of a linear injective operator 
L " C(M)  COe(M) or L "COe(M)  V««t(M) or L " C(S)  Vect(S). 
Denote by m  {0, 1.2 .... ;OE} the minimal m such that for anv s  0 the con- 
vergence to 0 of the sequence of sections L(f) in the C + topology implies the 
convergence to 0 of the sequence of hmctions f, in the C  topology. This means 
that the inverse to L behaves hot worse than a linear diffcrential operator of order 
m. Note that bv Theorem B1 we have m(Lx), m(L), and ,n(Lx.) 
In many cases the nmnbers m(Lx), m(LH) and m(Lx.H) are finite and can be 
found or estimated from above, sec examples below. Tracing the construction of 
the functions AI, BI and Ci in the proofs of Theorems 1.2 and 1.4. it is easy to 
check that if these numbers are finite, then: 
1. the C-closeness of AI to 1 holds provided that the 1-form 1 i8 close to 0 
in the C  topology with r = 2m(Lx) + 2: 
2. the C°-closeness of BI to 1 holds provided that the 1-form 
in the C  topology with r = 2m(Lx.H) + 2: 
3. the C°-closeness of Ci to 1 holds provided that the 1-form d 1 i8 close to '0 
in the C  topology with r = m(Lx,H) + m(LH) + 2. 
Therefore in Theorems 1.2 alld 1.4 the Coe-closeness of W'l to 0 can be replaced 
by the closeness in a weaker topology, and we obtain the following result. 
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Theorem B2. In Theorem 1.2 the C°-closeness of wl to wo can be replaced by 
the CT-closeness with r = 2m(L¥) + 2. In Theorem 1.4 the Coe-closeness of wl to 
wo can be replaced by the CT-closeness with 
r = lnax (2n(Lx.H) + 2. m(Lx.H) + n(LH) + 2). 
Examples (n = 2k). 1. If X has no singular points, then it is clear that m(L_¥) = 
0. Therefore the Coe-closeuess of w to w0 in Theorem 1.2 can be replaced by 
C2-closeness. Wc obta.in Theoreln 0.1. 
2. If the 1-jet of X vanishes at no points of the nmnifold, then it is easy to prove 
that m(Lx) _< 1. Therefore the Coe-closeness of w to w0 in Theorem 1.2 can be 
replaced tri" C4-closeness. 
Examples (n = 2/,: + 1). 1. If (co0) is a Martinet distribution, i.e.. dH(p) ¢: 0 
and X(p) ¢ 0 for an.v p  .b' = {H = 0}, then it is easy to prove that m(LH) _< 1 
and m(Lx.H) = 0. Therefore the C-closeness of CO 1 to CO0 in Theorem 1.4 can be 
replaced b.v C3-closeness. We obtain Theorem (/.2. 
2. Assume that dH(p) ¢= 0 for any p  S -- {H = 0}. Then the restriction of 
X t.o S is a. smooth vector field X]s on S. Assume that the 1-jet of Xs does hot 
vanish. In this case m(L) < 1 and m(Lx.H) <_ 1. Therefore the C-closeness of 
1 to COe in Theorem 1.4 can be replaced by C4-closeness. 
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WHEN ARE THE TANGENT SPHERE BUNDLES 
OF A RIEMANNIAN MANIFOLD REDUCIBLE? 

E. BOECKX 

ABSTRACT. \e deternfine ail RiemaImian manifolds for xvhich the tangent 
sphere bundles, equipped with the Sasaki metric, are local or global Riemann- 
ian product manifolds. 

1. INTRODI¢CTION 

When studying the geometry of a Riemmmian manifoM (3/. g), it is often useflfl 
to relate it to the properties of its unit tangeut sphere bmMle Tt3I. hl earlier 
work, we have been primarily interested in the geometric properties of T3I when 
equipped with the Sasaki metric gs. This is probat»ly the sinll)lest possible Rie- 
mamfian metric on T3I and it is completely determined by the nletric 9 on the 
base manifold ]lI. In this way, we bave obtained a. nUnlber of interesting charac- 
terizations of specific classes of Riemmmian manifolds. We refer to [2], [5]. [6], [7] 
and the references therein tbr examples of this. Also tangent sphere bundles T,.M 
with radius r different from 1 and equipped with the Sasaki metric bave been stud- 
ied recently ([9], [10]). The geometric properties of these Riemmmian manifolds 
may change with the radius. Sec [9] for an example of this. Of course, other 
tiemannian metrics on the tangent bundle and on the tangent sphere bundles 
are possible. Of these, the Cheeger-Gromoll metric 9cc may be the best known. 
However, for tangent sphere bundles, this specific metric yields nothing new. since 
(T,-M, gcc) is isometric to (T,./4i-4M, gs ). The isometry is given explicitly by 
p: Trl]l -- Tr/ lvq-l[: (X, t) v- (X, u/V + r2). 
It is a.n interesting geometric problem to determine when a tangent sphere lmn- 
dle, which we alwoEvs consider with the Sasaki metric in this paper, is reducible, 
i.e., when it is locallv or glol)ally isometric to a Riema.nnian product manifold. To 
our surprise, we could not find anv results in the literature concerning this ques- 
tion. Nevertheless, knowledge about reducibility could help to deal with geometric 
questions about, tangent sphere bundles, lu [4] for instance, we use it in an essen- 
tial way to determine all unit tangent sphere bundles that are semi-symmetric, i.e., 
for which the curvature tensor at each point is algebraically the saine as that of 
some symmetric space. Actually, that problem was the inspiration for the present 
article. As concerns the local reducibility of tangent sphere bundles, we prove here 
the following. 
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Local Theorem. A taTgent sphere bmdle (T,.]iI, gs), r > O. of a Riernannian 
manifold (M ', g).  >_ 2, is locally reducible if and mdy tf (M, g) bas a fiat factor. 
i.e.. (lI,g) is locally isometric to a product (M',g') × (llk, g0) where 1 _ k _ n and 
go denotes the standard Euclidean metric on II . 
The corresponding global version reads as follows: 
Global Theorem. Let (]il '.g). n >_ 3. be a Riernannian rnanifold and suppose 
that (TM. gs) is a global Rienmnnian product. Then. (]il, g) is either fiat or it is 
also a global Riemamia product, with a .fiat factor. 
Coversel.q, if(M, g) is a global product space (M', g') × (F , go) where 1 <_ k <_ n 
ad F is a co.nnectcd ad sin@ly connected .fiat space, then (TM, gs) is a global 
Riemannian product, also with (F, go) as a .fiat factor. 
In vicw of thc commcnts above, thcse results remain valid if we consider the 
tangent sphere bundles equipped with the Cheeger-Gromoll metric. 
This article is organized as follows. After giving the necessary definitious and 
formulas concerning tangent sphere lmndles, we show in Section 3 that onlv two 
types of decomposition for TM are possible: a vertical and a diagonal one. The 
special form of the curvature of (TrM, gs) for xrertical vectors is crucial here. In 
particular, the saine procedure does hot go through for the tangent bundle TM. 
Sectiou 4 deals with the diagonal case. We find that a diagonal decomposition 
gives rise to a Clifford representation via specific curvature operators. As a result. 
only base maififolds with dimension 2, 3, 4. 7 or 8 could possibly adinit diagonal 
decolnpositions. The diffelent dimensions are then handled separately. It turns out 
that diagonal decompositious can only be realized for a fiat. surface as base space. 
The general situation with a vertical decolnposition is treated in Section 5 and leads 
to the Local Theoreln above. The final section is devoted to global cousiderations. 

2. TANGENT SPHERE BUNDLES 

We first recall a few of the ba.sic facts and formulas about the tangent sphere bun- 
dles of a 1Riemannian manifold. A more elaborate exposition and further references 
can be found in [51 and [9]. 
The tangent bundle Tll of a Pdelnannian lnanifold (]il. g) consists of pairs (:r, u) 
where 3c is a point in M and u is a tangent vector to M at .r. The mapping 7r : TM -- 
M: (:r,u) - x is the natural projection from TM onto M. It is well known 
that the tangeut space to T]II at a point (a-, u) splits into the direct sure of the 
vertical subspace I'T]II(z,, 0 = kerr.l(,, 0 and the horizontal subspace HT]II(,, 0 
with respect to the Levi-Civita connection 
H T ]I I( ,, 0  
For w  T,]II. there exists a unique horizontal vector w   HTM(z., 0 for which 
7r.(w ) = w. It is called the horizontal lift of w to (x, u). There is also a unique 
vertical vector w   I'T]IIt.,O for which w(df) = w(f) for all fimctions f on ]iI. It 
is called the vertical lift of iv to (x, 
and HT]II(,, 0 and VT]II(,,O, respectively. Hence, everv tangent vector to T]II 
at (:r, u) can be written as the sure of a horizontal and a vertical lift of uniquely 
defined tangent vectors to ]Il at :r. The horizontal (respectively vertical) lift of 
a vector field X on M to TM is defined in the saine way bv lifting X pointwise. 
Further. if T is a tensor field of type (1, s) on M and X1,..., X-I are vector fields 
on ]il. then we deuote by T(X1 .....  ..... X_) " the vertical vector field on TM 
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which at (ce, w) takes the value T(Xlc,...,lv .... , Xs_ix) v, and sinfilarly for Ihe 
horizontal lift. In general, these are hot the vertical or horizontal lifts of a vector 
field on/ll. 
The Sasaki metric gs on T/iI is completely deIermined by 

9s(x h, ) = 9s(x v. ") = 9(x, ) o , 9s(x . v) = 0 

for vector fields X and }" on/il. 
Our interest lies iu the tangent sphere bundle T,./i1 of some positive radius r, 
which is a hypersurface of Tal consisting of ail tangent vectors lo (/il, g) of lenglh r. 
Il is given implicitly by the equation g,(«, u) = r 2. A unit normal vector field N 
to T/iI is given b.v the vertical vector field "/r. We see that horizontal lifts 
to (:r, u)  TM are tangent fo T/II, bul vertical lifts in general are not. For 
that reason, we define the tangential lift u , of u'  T.Cl to (x, u)  TI by 
1 
wt = wv - 7 g(w.u)N. Clearly, the tangent space lo T5I at (x,u) is spmmed 
bv horizontal and tmgential lifts of taugent veclors to M at x. One defines the 
tangential lift of a vectorfleld X on/iI in the obvious ,va.v. For the sake of notational 
clarity, we will use 2{" as a shorthmd for X -  g(X, u)u. Then .\' = .(v. Further, 
we denole by VT,.kI he (u - 1)-dimensional distribution of vertical tangent vectors 
to TrM. 
If we consider THI with the metric induced from the Sasaki metric 9s of T/i1, 
also denoted by 9s, we turn TrM iuto a Fliemannian manifold. Its Levi-Civita 
Colmection Ç is described completely by 

() 

1 
V x, I "t = o(I",,z)X t, 

Ç\.hY h 

: _ (H%A')y) h, 
2 
1 
= (Vxr)'+((,,.)x) h, 
- (Vx}')h I((X,}'))' 

for vector fields X and Y on/il. It.s Riemann curvature tensor/) is given bv 

(2) 

(X h, Yh)Zt 

(X h, Yh)zh 

for vector fields X, Y and Z on/il. (See [9].) 
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3. TWO TYPES OF DECOMPOSITION 
Let (/l.g) be a Fliemmmial mmlifold of dimension , k '2 and suppose that 
its tangent sphere bundle T,.M is (locally) reducible, i.e., (TM, 9s)  (/1,gl) x 
(M2, 9)- A point (.r, u) in TM corresponds to a couple (p. q)  I1 X 2lh, and 
the tangent space T(,,)TkI can be identified with TpI1 @ q.lI2. In the sequel, 
we will write T(,)kI and T(,)kI for TpI1 and TqM2, considered as subspaces 
of T(,)TM. in order not to make the notation too cumbersome. 
Suppose first that, at a point (.r, u) of TkI, the tangent space to one of the 
factors, say to I1, contains a nonzero vertical vector X t, X  TkI and X K u. 
Since we have a Riemannian product, the curvature operator (U, V) preserves 
the tangeut spaces to both factors for ail vectors U and V tangent to TtM. In 
partiçular, it follows that 
1 
(}", xt)x t = r  (9(.¥, X) }-' - g(X. }')X t ) C T(.u)3ll 
for all vcct.ors }  T«;I. As a consequence, I'T5I(.) C T(.u)M, and lI is at 
lcast (,- 1)-dimcnsional. Hencc, (f at a point 4E;l one of the factors contains a 
 nonz«v vertical ve«tor, if «ontains fb« complete vertical distribution al that point. 
 call the decomi)ositiou vertical at (a'. u) in such a situation. Note that this is 
the ce as soon  m{dim fil1, dira M2} > n. Indeed. if dira fill > , then 
- dim(l'E,lt,u ) + 
> (., - 1) +  - (2,, - 1) = 0. 
So, the only possibility %r the decomposition hot to be vertical at (z. u) is that 
dira I1 = , dim.12 =  - 1 {or conversely) and neither factor is tangent to a 
vertical vector. X call this a diagonal decomposition at (x, ). 
The major part of the sequel will be devoted to the diagonal case. Using a purely 
infinitesimal (i.e., pointwise) approach, we show that a diagonal decomposition is 
only possible in one specific situation. Afl.erwards. we study the case of a vertical 
decomposition. 

4. DIAGONAL DECOMPOSITION 

4.1. A suitable basis. In this section, we consider a diagonal decomposition 
Tr]l  1I1 x M.2 at (x, u) with dira/I1 = "n and dira/I2 =  - 1. For dimen- 
sional reasons, we bave 

dim(T(:,,)3ll  HT3I(,.,)) > O. 

Let Xn  TII be a unit vector such that Xn h is tangent to Il[ 1 at (x. u) and 
extend it to an orthonormal basis {X .... ,X,} of T«M. If rc.(,,,)(T(,,,)M1) ¢ 
TM. then there nmst be a vertical vector tangent to/I1 at (x. tt), contrary to the 
hypothesis. Hence, there exist well-defined vectors Y ..... }-t orthogonal to tt 
for which .\-h + tlt ..... Xn_l h ÷ t'n_l t and X, h are tangent to 1[ 1 at (x.u). 
Clearly, they form a basis for T(æ.)M. though uot in general an orthonormal one. 
Moreover. {Y1 ..... t;_, u} is a basis for TM too. Otherwise, there would exist 
a nonzero vector t"  TII, orthogonal to tt and to Yi, i = 1 .....  - 1. But then 
t "t would be orthogonal to X, h and to Xi h + E t, i = 1 .....  - 1. and hence 
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would belong to (Tla:,u)11Ii) - = T(a:,u)111.2, contrary to the hypothesis that kI2 has 
no vertical tangent vector. 
Next, consider the (n- 1) x (,,- 1) matrix a = (g,(I: ))i4= ..... _. Since 
this nmtrix is symmetric and positive definite, it cm be diagolmlized bv a suitable 
orthogonal transformation: 
p¢pt = diag(A2 .... , kn_l 2) 
where P = (Pij)  0(,- 1) and A, > 0 tbr i = 1 .... , - 1. Ifwe p/lt 
j=l j=l 
fo =  ..... - . thon oth {X5 .... 
or{honormal bases for T,M. Further. thc vectors 
2,  + ,L'=  ,, (a 
logether wi{h Xn  span the {angenl spa«e {o kI at (x. u) and these xwctors are 
pairwise orlhogonal. The tangent space to 
orthogonal vectors 
A,çi -«, i= 1 ..... t-1. 

Finally, we show that all the 
gs(/(U, V)W, T) = 0 at (x, u) 
to/111 and another one is tangent 
it follows that 
Using the expressions (2) for the 
condition 

numbers Ai are equal. To do this, we use that 
as soon as one of the vectors involved is tangent 
to M» In particular, for all i, j, k. l = 1 ..... n- 1. 

curvature tensor/ of (TII. gs), this lea.ds to the 

0=  (g((2»_L)-.fi)-g((,)-i)-ï»(.)-L)) ,,2 - 
Switching the indices i and j. as well  k and I, we find 
Using the symmetries of the curvature tensor, it then easily follows that Ai 2 = Ai 2 
Summarizing, we have 
Lemma. If Thl  ,I x M2 is a diagonal decomposition et (x, u) u,ith dira 5I = 
  and dimM2 = n - 1. then there exist orthonormal bases {X,...,X,} and 
is given by 
X1 h + A)] , .... X._  + A)_ t, X,, h 
and an orttogonal basis for T(z,u)M2 is given by 
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ReTnark 1. The number ) has a clear geolnetric meaning. Take a nonzero vertical 
rt--1 
vector U at (x, t)" U = i=  and a nonzero vector V tangent to M2 at (x, u): 
= i= fli (AXi h - t). The angle between thc two vectors h cosine given by 
«o() = - E 
By the Cauchv-Schwarz inequality, we bave 
1 < cos(U) < 1 
fi + A: - fi + A: 
with equality if and only if (a,,... ,a-l) and (fil,---,Ç-I) are proportional. 
conc]ude that the angle  t)etwccn lTrI(z,) and T(z.)II2 is such that cos 
1/ + A 2 or tan  = A. So, A determincs the angle between 1 "ThI and Iee at (x, 
(and hcnce also between I'T[ and hI at that point). 
Hemark 2. Actually, we can give a st.ronger formulation of the lemma. To see this, 
consi(tcr the mapl)ing " T¢.)M  I'TM(z,)" X h + l-t  l-t. Clearly, this 
ma.l)l)ing is linear an(l onto-onc on (Xh) .  restrict  to (X,,h)x  T(x.)SI 
and dcfine the linear mapping 
A" a   X x" Y  An.(x,)(n  ) 
whcre, as 1)cforc, " TrM  M is the natural projection map. Since 
.a =  .(.)(,-')')=  .(.)((.X',  + Y=*)/) = 
the map A is an isometry from u  to X . It associa.tes to a vector X, orthogonal 
to X, the unique vector ', orthogonal to u, such that Xh+ Al t is tangent to 
at (x, u) (or such that AX h - yt is tangent to ,I2 at (x, a)). So, in the lemma, 
we can actual]y choose an arbitrary orthonormal basis {X1,..., X-I} of X  (or, 
alternatively, an arbitrarv orthonormal basis {) ..... )_} of u).  will use 
this possibility in the subsequent subsections. The vectors X (up to sign) and u, 
on the other hand, are determined geometrical]y. 
4.2. Curvature conditions. Since (T,.M, gs) is a (local) Riemannian product, all 
the expressions of the form (U. V)W are zero when U is tangent to hl and W is 
tangent to Iee at (x, u). Using thc curvature fornmlas (2), this leads to a number 
of curvature conditions for the manifold M. Wc list some of these now. om now 
on, indices i. j, k and l belong to {1,... ,n - 1} unless stated otherwise. 
The tangential and horizont al components of (X h.  ) t) (AXkh _  t) give fise 
to 
2 
(3) n(.X', ÆX')Y) -  g(n(.X, X»), ) = 
while (X h, ,kÇ-h)(AX h -- tt) = 0 leads to 
4 
(5) 
(6) 
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Considering/(X/h + At'i t. }'jt)(/xkh -- t) : 0. We obtain 
2 
(7) 
4 
= 
(8) 2(vx,)(..))x+2(),)x+ (..))(.)x 
Finally, from (Xi h + t, Xjh)(Xk h _ t) = 0, we derive 
(9) 
2A  
These conditions can be rewritten in an casier form. To start, we take the inner 

product of (3) with t}. This gives 
2((.\',,, .\-) k;.,  ) 

This is equivalent to 

= -(t(, )t(, i)x,.\). 

(11) 2R(k-, t))X + R(,t))R(u.t))X, =-g(R(a. Yj)X,,R(u,t))X,)X.. 
By interchanging the indices j and l iii this expression and adding both formulas, 
respectively subtracting theln, we get 
(12) R(u,I-)R(u,I))X,, + 
= -2(n(. ? )x. (.  i)x,) x, 
(13) R(u, t)-)R(, l))X. - R(u. t ))R(u, t ))X. = 4R(I ). t-)X. 
Substituting (11) iii (4), we filld the simpler form 
(14) 2A(Vx.R)(u. 
Next, we substitute (3) in (5) to obtain 
Taking the inner product with t, we get 
= 
= 0 
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by (12). Hence, 
(15) (V.\-kt)(_¥n,.¥j)tt = 0 or. equivalently, 
Sul)st.itutiug (14) aud (15) lu (6), we find 

(16) 

(Vx.R)(. i)xn =0. 

1 
._ ( R(. Y )X,.. R( .. Y )X,,) 
= R(.\'n, Xj)X + 2(,,. (X., 
- (-,,. R(X» X).)X + 
lu order to rewrite (7). we proceed as with (3): we take the inner product with 

aud we use curvature properties to obtain 
4 
(1) 2R(,)X, + :(..Y):(,.'4)X =  (»Xç- X). 
(Note that we also ueed (11) to know that the left-hand side in (17) is orthogonal 
t.o X.) Again switching the iudices j and I and adding and subtracting the two 
fi)rmulas, we get 
4 
(19) R(u,))R(u.}})X,- R(u,}})R(u.))X, = 4R()},))X +  
SM)st.itutiug (]7) and (19) in (8), this rcduces to 

(2o) (v x,R)(, ))x = 
or equivalently, via (15), to 
(21) A(Tx, )(Xk, -\')u = 

 ,( - 1) 
r (&X - X), 

2( - - 1) 
r2 ((ik Yl -- (ilYk). 

It is now easilv verified that (9) is a consequence of the above formulas. As 
to (10), usiug (17) and (20), it simplifies to 
(2) 4EE(x. x)x + R(, (x, x-))x- (. (%., x))x 
In the rest of this section, we wi]] on]v need the Nrmu]as (12), (13), (16), (18), 
(19) and (22). 
4.3. Clifford structures. Putting j = 1 in (12) and (18), we see that 
4 
Since (, ) is a skew-symmetric operator, the nonzero eigenvalues of R(u. 
must bave even multiplicity. Hence, 
*if n is even, the eigenvahm -4/r 2 has even multipli«ity n - 2 on {Xj, X} . 
Hence. the eigenvalue corresponding to X must be zero. This implies that 
n**,Y))X = 0 for  =  .... ,, - . By (3), so n()),)X = 0 for 
j, k = 1 ..... n - 1.  conclude that X,, belongs to the nullitv distribution 



WHEN ARE TANGENT SPHERE BUNDLES REDUCIBLE? 2893 

of the curvature tensor R. |n this case, the conditions (12), (13) and (16) 
are trivially satisfied; 
if n is odd, the eigenvalue -4/r 2 has odd multii)licity n- 2 on {X, X,} ±. 
So, the eigenvalue corresl)onding to X, nmst be -4/r 2 as well. Hence, it 
follows that IR(u. }))X,I 2 = 4/r 2 for j = 1 ..... n - I. Bv Remark 2, we 
even have IR(u, Y)X,I 2 = 4/r 2 for every mfit vector 7t" orthogonal to u. Po- 
larizing this identity, we obtain 9(R(u. I')X,, R(u. Z)X,) = (4/r 2) 9(Y, Z) 
for all vectors V a.n(l Z orthogonal to u. In l)articular, the right-hand side 
of (12) equals -(85jt/r2)X,. In this case, conditions (12) and (13) are 
included in (18) and (19) if we allow the index i to I)e n. 

Next, we put i = j -¢ l in (18). Since R(u, 7t))Xj = 0 (this follows ri-oto 
R(u,I'))2Xj = 0), we obtain R(u.}'j)R(u,})Xj = (4/r2)Xt. Avplying the op- 
erator R(u, I)) on both sides, we have 

or, equivalently, 

Since the right-hand side of this expression vanishes both when n is odd and when 
 is even. we conclude 

(23) 

for j, l = 1 ..... n- 1. 
Ve are now ready to discover Clifford representations in our fornmlas, in partic- 
ular in (12) and (18). First, consider the case when n is even. For j = 1,...,  - 1, 
define the operators Ri acting on V" = T,]II by 

T 
R, = _ (, }i) - (x,, .)x, + (x, .)x, 

wllere (-,) = gc. In particular, it follows that R,Xi = \'n, RX = -Xi and 
RXj = (r/2)R(u, ;)X-, j ¢ i. Clearly, R, is a skew-symmetric operator and 
Ri 2 = -id. 
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For i ¢ j ¢ k ¢ i, we calculate: 
(Ri o Ri + Ri o Ri)Xn = -RiXj - RjXi 
=-ï-(R(u,Y)Xj+ R(u,}))Xi)=0. (by (23)) 
2 
(Ri c Ri A- Ri o Ri)Xi = Ri(/(u, })Xi) A- 
T 2 
= --f R(u, Yi)R(,Yj)Xi- Xj =0 (by (18)) 
 ,. r R(u, E)Xk) 
+ (,5){(,;-)X-g(R(,})X,Xj)X}) 
T 
4 
=0 (by (s) la (23)). 
So, for i,j = 1,..., - 1. the operators R, satisfy 
R o R + R o R, = -2 iid 

and they correspond to a Clifford representation of an (n - 1)-dimensional Clifford 
algebra on an n-dimensional vector space. 
It is well known (see, e.g., [1] or [3]) that a given real Clifford algebra, say of 
dilnension m, has onlv one (if m - 3 (mod 4)) or two (if m =- 3 (mod 4)) irreducible 
representations and that the dimension n0 of the corresponding irreducible Çlifford 
module is completely determined by m. This relationship is given in the following 
table. 

rn 8p 8p+l 8p+2 8p+3 8p+4 8p+5 8p+6 8p+7 

no 2 4p 2 4p+I 2 4p+2 2 4p+2 2 4p+3 2 4p+3 2 4p+3 2 4p+3 

For a reducible Clifford module, the dimension is a multiple kno of the number n0 
corresponding to the appropriate Çlifford algebra. 
In the present situation, we bave m = n- 1 and kno = t for even . Therefore: 
 if  = 8p: 8p = k24p-  and hence p = 1, k = 1 and n = 8; 
 if n = 8p + 2: 8p + 2 = k24p+l and hence p = 0, k = 1 and n = 2; 
 if  = 8p + 4: 8p + 4 = k2 4p+2 and hence p = 0, k = 1 and n =- 4; 
 if  = 8p + 6: 8p + 6 = k24p+a, which has no solutions. 
Next, suppose that  is odd. Now, we define operators Ri, i = 1 ..... n - 1, 
acting on V n+ = TxII  NXo 

T 
R; =  n(, y)- (Xo, .)x, + (x,,-)x0 
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where ( -, .) = .q. @ go with go(aXo, bXo) = ab. Precisely as before, we show that 
1 i o lj + Rj o 1 i = -25ia id for i, j = 1, .... n - 1. So, we bave again a Clifford 
representation, this time with m = n - 1 aud/,"n0 = n + 1 for odd n. Therefore, by 
the table al)ove: 
 if n = 8p + 1: 8p + 2 = k2 4v altd hence p = 0. k = 2 and n = 1: 
 if,=Sp+3:8p+4=k2 4v+2andhencep=0,k=land=3: 
 if n = 8p + 5: 8p + 6 = k2 4v+a, which has no solutions: 
 if  = 81»+ 7: 8p + 8 =/,'2 4v+a and hence p = Il. k = 1 and  = 7. 
We conclude from this subsection that diagonal decompositious can only occur 
when the base manifold has dimension 2.3, 4, î or 8. (The case n = 1 is irrelevant, 
since then T,.M has dimension equal to one and no decoinpositions exist.) 

4.4. The relnaining dimensions. 

Case n = 2. In this situation, we have a two-dimensional manifold for whiçh the 
mfllity vector space of the curvature tensor is non-trivial. This implies that the 
curvature tensor is identically zero and the space is fiat. 
Conversely, sin«e any tangeut sphere tmndle of a fiat surface M2(0) is a fiat three- 
dimensi(mal space, a diagonal decomposition actually exists around each point (x. u) 
of TM'(O). Note, however, that we also bave T]II2(O) - M")(O) x S(r) with 
{x} x S(r) _ a--l(x). So. TM'(O) also admits a vertical decomposition. 

Case n = 3. Let Xa 1)e the unique unit vector (Ul) to sign) such that Xa h is 
tangent to kl at {x, u). Pick a Ulfit vector X orthogonal to Xa and let  be the 
corresponding unit vector orthogonal to v {i.e., .¥h +A)-t is tangent to .iii). l¥oln 
the comments at the beginlfing of Subsection 4.3. we know that (r/2)R(u.))Xa 
is a unit vector, which is moreover orthogonal to X1 and Xa. So, we obtain an 
orthonormal basis {Xt,X2, Xa} by defining X2 to be X: := (r/2)R(u.'a)Xa. Let 
t) be the corresponding unit vector orthogolml to u and 1". (Since each 1 is fixed 
together with its correspouding Xi, we will hot luentioli this explicitly anvlnore in 
what follows.) 
Using the properties of the operators R( u, ') and (v. ), we then deduce that 

(24) R(u, Y1 )x = 0, R(u, Y )x = - -. x3, 
/(, ?)x, = 2_ xa, R(u, ?)x2 = o, 

and from (13) and (19) it follows that 

R(tt "1.)X 3 - 2_ 

Next, we COlnpute R(X, Xj)X, i, j. k = 1.2.3, ffonl the equalities (16) and (22). 
writing R(u,R(Xi, X)u)Xa as Y'9(R(u, Yz)Xi, Xj) R(u,})Xt. and using (24)and 
(25). This gives 

(26) 

1.2 R( X1, 
1. R(X,, X3) 
1.2 R(X2, X3) 

Xa X2 Xa 

-AX AX 0 
-CX3 0 CX 
0 -CXa CXa 

where A = (,4 _ ,2 q_ 1)/),2 and C = (aA 2 + 1)/ 2. 

(25) t(Y,)xl=-x2, (,)x.=xt, (Y,)xa=o. 
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Since both {X1, X2, X3} and {¥], t), u/r} are orthonornial bases for Txll, there 
is an orthogona] matrix Q = (qij) 6 0(3) such that 

X., = Q Y.,  

Changing Xa to -Xa if necessary, we ma,y even sut»pose that Q 6 SO(3). Then 

R(Xt, A'3) 

= (qlqa2 -- qt2q31) H(}], }')) + qllq33 --ql3q31 
?- 
+ q12q33- q13q32 R(}), 
= -q3 R(Y1, t;_) - q2___ R(u, ti) + q__l R(u, 
T F 

If we let both sides act. on X1, ¥._, and X3 and if we use (24), (25) and (26), we 
find that 

q21 = -C/2. q2 =0, q3 = 0. 

Since Q  S0(3), it follows that qe2 +qe22 ÷q23 2 = 1 and hence 1 = (3A2+ 1)/2A 2 
or A - + 1 = O. which is a contradiction. Hence, no three-dimensional manifold 
adnlits a diagonal decomposition of its tangent sphere bundles at anv point. 

Case 11 = 4. Let X4 be the unique unit vector (up to sign) in the nullitv distribu- 
tion of R,. Take two mutuallv orthogonal unit vectors X and X.,,_ perpendicular 
to X4. Since (r/2)R(u, Y1)-\'2 is a unit vector and orthogonal to X1, X2 and X4, we 
can define X3 := (r/2)R(u, t])X2. From the properties of file operators R(u. 
i = 1,2, 3, it follows that 

(27) 

r R(,,, "1) 
," R(u, t&) 
,'R(u, t5) 

X1 X2 Xa X4 

0 2Xa -2X.2 0 
-2X3 0 2X 0 
2X -2X 0 0 

Next, we decompose Xa with respect to the basis {tq, t.), t, u/r}: 

X4 = qt] + q2I') + q3} + qa -, 

q + q2  + q32 + q42 = 1. 

Then R(u, Xa) = ql R(, }'1) ÷ q R(u, Y.,_) + q3 R(zt. }5)- Since Xa belongs to the 
mfllity distribution of R, this operator vanishes identically. By (27), we nmst bave 
q = q2 = q3 = 0. Hence, Xa = :t:u/r. But this is impossible, since u clea.rly does 
hot belong to the nullity distribution. So, also for four-dimensional mallifolds, a 
diagonal deconlposition of its tangent sphere bundles does hot exist at anv point. 

Case n = 7. The argtlnlent for n = 7 goes along the Saille lines as that for n = 3. 
but it is more involved technicallv. Again we start with the trait vector Xr, uniquely 
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deternfined up to sign. su«h that Xr h is ta.lgent to 2I, and with an arbitrary unit 
vector X, orthogonal to X7. The unit vector X2 := (r/2)R(u, }',)X7 is orthogonal 
to both X aud X7. Then it follows t.hat 

R(, ;)x, = o, 
R(u, }))X = - Xr, 
,ï, 

R(u. }')X2 - 2 X7, 
R(.u. )x2 = o, 

R(-u, Y)X7 = - X2, 
R(u, ')Xr -  " 
-- -TX. 

Note that R(u. Y1) aud R(u, t?) preserve spal,{Xl,-\'2, X7}, hence by skew-sym- 
metry also its orthogonal complemeut. Next, take a unit vector X4 orthogonal 
to X1, X.2, Xr and define the unit vectors Xs := (r/2)R(u,t))XE aud X := 
(r/2)R(u, t'I)X4. Then Xs and X are ah'eadv orthogoual to X, X2, X4 and Xr. 
Further, 

£2 
i-2 
-- g(R(u,}])R(u,}'2)X4, X4) 
4 
£2 
(28) = g(R(u.I))R(u,}i)X4,X4) (by (18)) 
r 2 
-- g(R(u, })X4, R(,u, )X4) 
4 
= -{Xs, 

and X5 and X6 are mutually orthogonal as well. Finally, sin«e R(a, ]q)X5 is or- 
thogoual to X, X2, Xs, Xr and 

we mav defil,e X3 := (r/2)R(u, I'I)X. 
In this way, we have defined ml orthonormal basis {X1 ..... Xr}, and the actionb 
of the operators R(u, }), i = 1 ..... 6, can be COUlputed explicitly in this basis using 
the properties (12), (18) and (2a) above. We obtain 

 r R(,, \) 
 r R(u, 
" R(-u, 
 r R(u. 
r R(-. 

X X.2 X3 X4 X5 X6 Xr 

0 -2Xr -2X5 2X6 2X3 -2X4 2X2 
2Xï 0 2¥6 2X5 -2X4 -2X3 -2X1 
2X5 -2X6 0 -2X7 -2Xt 23£2 2X4 
-2X6 -2X5 2X7 0 2X2 221 -2X3 
-- 2X3 2X4 2X1 - 2X2 0 - 2X7 2X6 
2X4 2Xa -2X2 -2X1 2Xr 0 -2X5 
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Next, ve calculate R(}]-, }-)Xk from (13) and (19): 

X1 X2 23 X4 X5 26 X7 

--222 2X 2X4 -2Xa 226 -2X5 0 
-2Xa -224 22 222 0 2XEE 
-2X4 223 -222 2X1 227 0 -2X5 
-225 -2X6 0 -227 221 222 224 
-226 2X5 -2XEE 0 -222 221 2Xa 
224 -2Xa 222 -22 227 0 -225 
-223 -2-\'4 221 222 0 -227 2\'6 
2X6 -225 -2X7 0 22 -22 223 
-225 -226 0 227 22 222 -2Xa 
2X2 -22 -224 223 226 -225 0 
0 227 -2-¥5 -226 223 224 -222 
227 0 -226 2X5 -224 223 -2XI 
227 0 226 -225 224 -223 -2XI 
0 -227 -225 -226 2X3 224 
222 -2Xt 224 -223 -226 225 0 

Using (16) and (22), ve ca,, now compute the «,r'«ature components R(Xi, 2j)Xk 
for i,j,k = 1 ..... 7: 

r 2 R(X 
r 2 R(X1 
r 2 R(X 
r 2 R(X1 
r 2 R(X1 
r 2 R(X1 
r 2 R(X2, 
r 2 R(X2, 
r 2 R(X2, 
r 2 R(X2, 
r 2 R(X2, 
r 2 R(X3, 
r 2 R(X3, 
r 2 R(X3, 
r 2 R(X4, 
r 2 R(X4, 
,,.2 R(X4, 
r 2 R(Xs, 
r 2R(X6, 

X1 22 X3 X4 X5 X6 X7 

-A22 AX 224 -223 226 -225 0 
- B23 24 BX - 22 0 0 0 
-B24 -X 3 22 BX 1 0 0 0 
-BX5 X6 0 0 BX -X2 0 
-BX6 -X5 0 0 22 BX 0 
-CX7 0 0 0 0 0 CX 
-X4 -BX3 BX2 X 0 0 0 
23 -BX4 -Xt BX2 0 0 0 
-X6 -BX5 0 0 BX.2 X 0 
25 -BX6 0 0 -X BX2 0 
0 -ex 7 0 0 0 0 C. 2 
222 --221 -AN4 A23 226 -2X5 0 
0 0 - BX5 26 BX3 -X4 0 
0 0 -BX6 -25 24 BX3 0 
0 0 -CX 7 0 0 0 CX 3 
0 0 -X6 -BÆ5 B24 X 3 0 
0 0 25 -BX6 -23 BX4 0 
0 0 0 -CX7 0 0 CX_ 
222 -2X 2X4 -223 -AX6 AX5 0 
0 0 0 0 -CX7 0 CX5 
0 0 0 0 0 -CX7 CX6 

where A = (A 4 - A 2 + 1)A 2, B = (A 2 + 1)2/A 2 and C = (3A 2 A- 1)/ 2. 
We now show that the tables above are incompatible. To see this, we relate 
the two orthonormal bases {X ..... 27} and {} ..... Y6, u/r} by an orthogonal 



WHEN ARE TANGENT SPHERE BUNDLES REDUC1BLE? 2899 

transformation. Let Q = (qij)  0(7) be such that 
Putting Q, := qi.qj - qqj, we thon have the equality 
6 6 
ij 
k<l=l k=l 

No, 

and 

2 

= r g(R(X,X2)X*,X4) 
6 6 
k</=l k=l 

= (QI - Qàï + Q) 

u)X3, X) 

12 
2 : .2 g(I(XI,X2)X5,X6) _ 2(Q22 ÷ Q3142 _ Q56)- 
This implies that Q] = 1. Now, using the Cauchy-Schwarz inequality and the fact 
that Q is orthogonal, we find that 
1 = QI = qllq2-. - 
Hence, ail the inequalities must be equalities. In particular, we have q13 .... 
ql7 = q23 .... = q27 = 0 and consequently 
X =cosO l +sinO )), X.2 =q(-sinO1Yl +cosO I) 
where ci = :t:1 and 01 is some real number. In a similar way, we can show that 

Q = Qs566 = 1 and that 
X 3 --- COS 0 2 }] ÷ sin 02 
X5 = cos03 t' + sin03 Y6, 

X4 = e2(-sin02 l + cos02 
X6 = 3(- sin 03 } ÷ cos03 Y6)- 

As a consequence, we also have X7 = cuir, e = -1-1. Using the tables above, we 

find that 
0 

= r e R(X, XT)X3 = -e (cos0 r R(u, }'1)X3 ÷ sin01 r R(u, 
= 2e(COS01Xs - sin01 X6), 
which gives a contradiction. So, also seven-dimensional manifolds cannot bave a 
diagonal decomposition for their tangent sphere bundles at anv point. 

Case n = 8. This case is treated as the case n = 4, but the appropriate choice for 
the basis {X1,..., Xs} requires a little more tare. Let Xs be the unique unit vector 
(up to sign) in the nullity distribution of R and take two arbitrary unit vectors 
X and X2 that are mutually orthogonal and perpendicular to Xs. As before, we 
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define X3 := (r/2)R0*, })X2, which is a unit vector orthogolml to X1, X2 and 
It follows that R(u. }))Xs = 0 for i = 1.2, 3, and 

n(.. }i)xl =0. 
R(U. l.))X1 -- 2 3, 
R(u, Ya)X1 = 2_ 

Because they are skew-symmetric, the operators R(u, }i), H(u,}:z) and R(u.};) 
also preserve l I = { X 1, X2, Xa, Xs } ±, and on t his space t hey ant i-conmmt e by (18). 
It is easv to check that the operator (ra/8)R(u, I])R(u, I))R(u. }5) is symmetric 
on II" and that it squares to the identity on Il . Hence, it bas a basis of eigenvectors 
corresponding to the eigenvalues +1 and -1. Let Xa be a unit vector in Il" such 
that r a R(u, Yt)R(u, Yz)-/ï'(u, }5)Xa = 8eX4 where e = +1, and define 

r/(. };)X. 

Clearly, X5, X0 and X7 are unit vectors orthogonal to X1, X2, X3. X4 and Xs. 
A comlmtation similar to (2) shows that they are also orthogonal to one another. 
So, we have an orthonornml t)asis {X1,...,Xs} for T3I. It is now possible to 
compute explicitly the action of R(u. ), i = 1 ..... 7. from the condition (18). VVe 
get 

 /(, i) 
r R(u, Y2) 
-/(u.}) 
.,-/(.. ;) 

X X2 X3 X4 X5 X6 X7 Xs 

0 2-\'3 -2X2 2X5 -2Xa -2eX7 2e-\'6 0 
-2X3 0 2X 2N6 2eX7 -2Xa -2eX5 0 
2X2 -2X1 0 2X7 -2eX6 2eX5 -2X« 0 
--2X5 -2X6 -2X7 0 2XI 2X2 2X3 0 
2X4 --2¢-\'7 2cA'6 -2.Y1 0 -2e-¥3 2e-\'2 0 
2eXz 2Xa -2eX5 -2X2 2eXa 0 --2eX1 0 
-2eX6 2eX5 2X4 -2Xa -2eX 2eX1 0 0 

Next, decolnpose X8 with respect to the basis {I],..., I), u/r}: 

Xs=qll]+---+qE})+qs-. ql 2+.--+qs  = 1. 

Since X8 belongs to the nnllity distribution of the curvature, we have 

7 
o= t(,x) = , q t(.) 

and ffom the table above we deduce ql = .... q7 = 0. Hence. XS = Tu/r, but 
this is impossible since u does hot belong to the nullity distribution. So, also in 
the eight-dimensional case, a diagonal decolnposition of the tangent sphere bundles 
does not exist at even a single point. 

Remark 3. The operator r 3/(u. ]/)]:(u,})/(u. Y3) acts as 8eid on the vector 
space spanned by X4, X5, X6and Xr, as is seen easilv froln the previous table. 
The two different cases, e = +1 and e = -1, correspond to the two non-equivalent 
irreducible Clifford representations of the seven-dimensional Clifford algebra. 
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5. VERTICAL DECOMPOSITION 

Now, we sui)pose that we have a vertical decolnposition T,.III - III1 × 3I such 
that I'T,.I(,) C T(,)I cverywhere. In this situation, if (x. u)  ,ll  {q} 
for SOlne q  M2, then -(x) C I x {q}. Çonsequently, we bave AI l X {q} = 
--1(7(/l  {q})). SO, th lçve [1  {q}, ('orrspolldllg to th prodllçt, project 
under  to a foliation L on (M.g) and n-(L) = {Mt x {q},q  Al2}. Let 
L be the distl'i]mtioli Oli Al tangent to L. I)cfinc the distribntion L2 to bc the 
orthogonal distrit)ution to Lt on I. Then 
where h denotes lhe horizontal lift. 
If X and ]" are vector fields on I tangent to L and if, i are tangent to L2, 
then X h, yh are tangent to 1 and rh, V h are tangcnt to I2. Because of the 
product strncture, we have that çx - and çg X  arc t311gcllt to [1 lld Çg i h 
and Ç.x' Un are tangent to AI2. Using the expressions (1) for Ç, this means that 
 Ç.x]" and ÇuX are sections of L" so, L is totally geodcsic and even 
totally parallcl: 
 ÇUl" raid ÇA-U arc sections of L2: so, also L2 is totally geodesic and totally 
parallel (iii part icular, L is integrable with sociated foliation L2); 
 R(U, V)u = R(X, U)u = 0: so, L2 is containcd il the nullitv distribution 
of the curvature. The leaves of L2 are therefore fiat. 
These properties imply that L and L.2 consist of the leaves of a local ielnamiian 
prodnct M  M' x N  where k = diln L  n (sec [8]). 
Suppose converselv that I  is locally isometric to M' x N  for 1  " 5 n. This 
gives rise to two foliations on I: L = {M'x {v}. t, G N  } and L2 = {{p} xN,p G 
I}. Define two COlnplelnentary distributions L and 2 on TI by 
L = l'Trl  h(TM'). L = h(TNk). 
It is easily checked sing (1) that  and {2 are totally geodesic and totally parallel 
complementary distributions. Hcnce, the leaves of their correspollding foliations È 
and  are actually the leaves of a local Riemanniall product, hl particular, note 
that  = {-(3I' x {t'}),v  Na}. So. TM is indeed locally reducible. 

6. (LOBAL RESULTS 

YVe continue with the notation of the previous seçtioll. In order to derive results 
concerlling the global reducibility of (TriiI, gs ), we will exploit the relationship 
between the foliations L1 and L2 of (il1, g) and the foliations t and 2 of (Tr]II. gs) 
iii t.he case of a vertical deconposition. "lVe ha.ve alreadv relnarked that L1 and 1 
deternfine each or her reciprocally by L 1 = re (! 1 ) and   = re- l L 1- The relat ionship 
between the foliations L2 and _ is hot so straightforward. We still have L = 
rc(Ï2), but deternfining L2 from L requires a little more tare. To construct the 
leaf ç of 2 through a point (x, v)  T,.III, consider all the curves iii the leaf S 
of L_ starting at z  III. Then, ç consists of all end-points of the horizontal lifts 
of these curves starting at (x, u). We call ç the horizontal lift, of S through (x, u). 
Since  is evervwhere horizontal, the lnap re-  -- S is a local isolnetrv and , is 
a Riemalmian covering of S. \¥hen S is silnply COlmected. 5} and S are globally 
isolnetric and. iii particular, one-to-one. 
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\Vith these comments in nfind, we now proceed to the proof of the Global The- 
orem. So, we suppose that dira M  3 and that (T,.M. gs) is isometric to a global 
Riemalmian product (kl, g) x {M2, g2). Since dira M  3, this is a vertical decon> 
position and I'T,.M is tangent to one of the fmtors, say M. Cnsider kl and 
as submanifolds of TAI {i.e., choose one leaf of eh of the foliations L and 
and define the submanifolds M' := (kl) and F := (M2) of M. From the local 
considerations in the previous section, we know that (M.g) is locally isometric to 
the Riemannian product M' x F and that F is fiat. 
We show that there is a onc-to-one correspondence between M and M' x F. Take 
a. point x  kl and consider an arbitrary vector tt E Tkl of length r. Through 
(.. u) e Tal, there is a unique leaf = of L and a unique leaf 2 of L2- Because 
of the product structure on T,.M,  cuts Al2 in a unique point O  A& and 
 cuts al, in a torique point   al,. Put p := ()  ai' and q= := (0)  
 claire that lhe con'espondcnce kl  al' x F" ,,"  (v=, q) is well-defined, i.e., 
independent of the choice of the ta.ngent vector u. To ste this, take another vector 
,,  Tal ,f length r. Sin«e (, u) = a" = (.r. t,), the leaf  of 1 contains both 
(.r, t,) and (,r, v); so we have u = ,'1, 0, = 0,. and q,, = q,,. The unique leaf 
of 2 through (a', t') is different from ç2.- However, both are horizontal lifts of 
S2, = w(2u). So, if #(t) = (.r(t),u(t)) is a curve in ç2u such that (0) = 
and (1) =,,  ,/I, then 7 = ° runs flom,r tot  M in $2,. Denote 
by  the horizontal lift of 7 starting al (a',v). Çlearly,  lies in 2 and ends 
On the other hand. starting from a couple (p. q)  M' x F, we find the corre- 
sponding point ,r  al as  = u(, 0) for some   al with () = p and the 
unique 0  M.e with (0) = q- Via an argmnent as above, one shows that  does 
hot depend on the choice of  and that the nmp (p, q)  defined in this way is 
the inverse of the map   (p, q). 
Next, we note that the correspondence kl  M' x F:   (p,q) is defined 
so as to respect the local product structure. In particular, the metric  on 
corresponds to the product metric of M' x F, and the first statement is proved. 
Conversely, suppose that (M, ) is the global product space (M', ') x (F, g0)- 
By choosing a leaf of both product foliations, one can consider M' and F  sub- 
manifolds of M. Let «0 be their intersection point and choose a vector uo  ToM 
of length r. Define kl as the inverse ilnage of M' under the projection  and 
k12  the horizontal lift of F through (a'0, u0). Since we suppose F to be simply 
connected, kl. is isometric to the fiat space (F. 0) and ,1 and I2 have {0. 
 unique intersection point. 
We show that there is a one-to-one correspondence between TkI and kI x 
Take {, u)  TM and denote by S1 the unique leaf of L 1 Oll al through m and by 
$2 the unique leaf of L2 on kl through  Then, the leaf  of  through ( 
is given by -(S) and the leaf 2 of 2 through (, u) is the horizontal lift of 
through this point.  cuts a& in a unique point O with (0) = S  F, and 
2 cuts kl in a unique point  with () = $2  al'. (Note that the simply 
connectedness of F is essential to ensure uniqueness.) Çlearly, the correspondence 
Tkl  1 x M2" (, u)  (, 0) is well-defined and it is hot dicult to construct 
it.s inverse. Since this correspondence also respects the local product structure, the 
metric gs on TM corresponds to the product metric on kI x k&. This completes 
the proof of the Global Theorem. 
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Remark 4. The proof of the Global Theorem continues to hold when n = 2 for the 
case of a vertical global decomposition of (TAI, gs). Clearly, the base manifold is 
then fiat.. That we need the simply commctedness of the fiat factor tan be seen 
from the example of a two-dimensional fiat cone C The vertical and horizontal 
distributions on TrC are both integrable, and locally their integral manifolds are 
the leaves of the local product foliation on T,.C If it were a global decomposition, 
everv maxinml integral manifokt of the horizontal distribution would intersect ev- 
ery vertical fiber exactl.v once and it would be isometric to C under the natural 
projection 7r. This would define a global parallelization of C, contrary to the fact 
that its full holonomy group is non-trivial. 
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CRITERIA FOR LARGE DEVIATIONS 

HENRI COMMAN 

ABSTRACT. V'e give the general variational form of 
lira sup(./i, e h( x ) / t tta ( dx ) ) t 
foi" any bounded above Borel measurable function h on a topological space X, 
where (tt) is a net of Borel probability measures on X, and (ta) a net in ]0. oe[ 
converging to 0. When X is normal, we obtain a criterion in order to have a 
limit in the above expression fol" ail h continuous bounded, and deduce new 
criteria of a large deviation principle with hOt necessarily tight rate functiom 
this allows us to remove t he tightness hypothesis in various classical theorems. 

1. INTRODUCTION 

Let Cb(X) be the set of real-valued bounded continuous functions on a topological 
space X, (Va) a uet of Borel probability measures O11 .', and (/a) a net in ]0. 
converging to 0. For each [-o% +ec]-vahled Borel measurable function h on X. 
we write ptâ (eh/t) for (fx eh(x)/t° ta(d'r))t«" and define A(h) = log lira pta" (e h/t) 
provided the limit exists. 
The ailn of this paper is to clarify the relation between the existence of A(h) for 
all h  Cb(X). and the one of a large deviation principle for (p) with powers (t). 
This problenl origillates fronl Varadhan's theorenl, which sta.tes that if X is regular. 
then such a prillciple with tigllt rate function .1 inlplies the existence of A(h) for all 
[-oc, +oc[-valued contilmous fimctions h on X satisfying some rail condition (in 
particular for ail h  rb(X)), with moreover A(b) = sup:ex{h(x ) - J(:r)}. This 
theorem is a crucial argument in the proofs of all related results: in particular, these 
also hold under some tightness hypothesis (of the rate fimctiom or of the net (it2 
i.e., exponential tightuess). 
We present here a new approach based on a variational representation of the 
fuuctional lim suppt2(e'/t), and a criteriou of existence of A(-) on Cb(X) wheu 
X is uormal. This leads to functional as well as set-theoretic large deviation cri- 
teria, which allow us to relnOVe the tightness condition in various basic results of 
the theory; moreover, the proofs are nonstandard since there are no compactness 
arguments in the entire paper. Notice that all the results work for general nets of 
measures and powers, and (except for Section 4 and "(i)  (il)'" in Theoreln 3.3 
where X is assumed to be normal) for a general topological states space. 
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We begin in Section 2 bv stating somewhat unusual equivalent definitions of a 
large deviation principle (Proposition 2.3) which imply the existence of a minimal 
rate flmction. 
In Section 3. we prove a general variational form of lim supp « (e h/t« 1y) for any 
Bord measurable flmction h on X and any Borel set }" C X sa.tisving the (local- 
ized) tail condition of Varadhan's theorem (Theorem 3.1). Bv the saine methods. 
we obtain a sucient (and necessary if X is normal) condition for the existence of 
A(h) for ail h  Ch(X), in the spirit of the Port manteau theorem (Theorem a.a). 
A generalized version of Varadhan's theorem without anv tightness assumption is 
a direct consequence (Corollary 3.4). 
In Section 4, assmning that X is nornml, we look for necessary and sufficient con- 
ditions that A(-) (as a flmctional on Ch(X)) nmst satisfv in order to bave a large de- 
viation principle. This is obtained in Theorem 4.1. which gives rive such conditions: 
in particular, it states that large deviations occur if and only if lin A(hi) = A(h) for 
each increasing net (h) in C(X] converging pointwise to h  C(X). As corollaries, 
va.rions basiç results hold verhatim without tightness ssumptions: this is the ce 
for the eqnivalence imtween the Laplace and large deviation principles (Crollary 
4.2), and for thc variational form of a rate flmction (4.5): a large deviation princi- 
ple is characterized  a convergence in a mrrow space of set-fllnctions much larger 
than capa.cities (Remark 4.5]: Crollary 4.3 improves Brvc's theorem by weakening 
the exponential tightness lgpothesis; (4.2) gives the infimum of a rate flnction 
J on anv closed set i,, terres of lira inf p" (-) (resp. lira sup p" (-)), generalizing a 
well-known expression of J. 

"2. PREL1MINARIES 
Let .T" (resp. Ç) denotes the set of closed (resp. open) subsets of X. For each 
O 
[0. +]-valued function f on X we denote by ] (resp. f) the least upper semi- 
continuous function on X greater than f (resp. the greatest lower semi-continuous 
function on X less tlmn f), and define a map 7I " 9 -- [0.+oo] by 7I(G) = 
sup.eG f(x) for ail G G {7. 
We collect here some characterizations of positive bounded upper senfi-contim- 
ous flmctions that we will use in the sequel. 
Lemma 2.1. Tbere is a bijection between the set of positive bounded upper semi- 
«ovtinuous fvnctios f on X and the set of maps ï " Ç - [0. +OE[ satisfying 
(9.1) 7(OGi ) =- sup',/(Ci) for all {Gi:i  I} C g, 
given by the maps 
 7  tir(x) = infeç,=e 7(G) for all x  X. 
 f-*7EEf. 
Mor«ov«r, for ch positive bound«d fun«tion f on X. the followin properties hold: 
(i) f = fæ; 
(ii) f is the unique positive upper semi-continuous function h on \" satisfying 
(iii) sup: c f(x) = infG»:Gç Tf(G) for all  C X: 
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Proof. Let f be a positive bounded upper semi-continuous flmction on X. For each 
1'" C X we ha.ve supr f _< Sllp{jï<sup. f+e} f  Sllpt f ÷ 6; since {f < supr f + e} 
is open and contains }', we obtain 

(2.2) sup f = inf 
 GD;Gç 

whence f's = f" \Ve now prove that %% = "y for ail 7" ç --' [0, +oo[ satisfying (2.1); 
let 7 be such a map. , > 0. and dcfine G), = U {G E 
and for all x E G, we have f.c(x) <_ "y(G) <_ u so that [,J<a G,, C {f. < ,}. For ail 
aï  {f. < ,} there is G 9 .c such that "y(G) < ,, and thus 
This shows that {f. < A} C QJ,«a G, and thus {tir < 3,} = QJ«a G, which is an 
open set, so that f. is upper semi-contilmous. Let G 
(with the convention supo = 0). If supa f < 3(G), then G C {f < 
for some e > 0. Since ri is upper Selni-continuons, {f < 7(G) -e} is open 
with {f < 3(G) -e} C G(a)_, and since 7 is clearly ilwreasing, we obtain 
() N 7({f < ()-«})  (G(a)-) N ()-«, which gives the contradictiou. 
Thus 7(G) = sup G f7 for all G  Ç and the first assertion is proved. 
Let f be a positive bomided functioli on .¥. Then 7I is bomMed and satisfies 
(2.1), so that f, is upper semi-contimloUS with fT,  f. For ail positive bomded 
upper semi-contimous fllnctions f  f, we have I 
f, which implies f = f. This prove (i). If h is a positive upper Senli-Colitinuous 
flmction on X satis[ving % = 7I, then h is bounded and (il) follows ri'oto (i). Let 
Y C X. By (2.2) we have sup. f = infGDY.Geç suPG f. and since  = I by (ii). 
(iii) holds. Let h = V{9  [0. +[x: % = YI}- It is easv to see that % = f, and 
since fh is upper semi-contimtous with fh  h. we have Yh = fh = f whence 
h: fTh = f by (i). Thus (iv) holds. 

Definition 2.2. We say that (p) sa.tisfies a large deviation principle with powers 
(t) if there is a lower semi-continuous flmction .1 : X -- [0. :c] such that 
lira sup  (F) _< sup e -d (x) <_ sup e- a (x) _< lira inf l,t« (G) 
xF xG 
for all F  9 r, G  {ï with F C G. Them J is called a rate function for (#t«), which 
is said to be tight if it has colnpact level sets. 
Notice that in the literature, a large deviation principle is in general defined for 
 1/n 
nets (lt)e>0 or sequences t#, ,eN*- In the sequel, when we will refer to known 
results that will be proved again, we will hot make this distinction and state them 
with general nets of measures and powers. 
By (2.3), the following proposition shows that the set. of rate functions for 
bas a minimal element: it is the only Olm if X is regular since it is well known ([2], 
Lemma 4.1.4). 
Proposition 2.3. The following statements are equivalent: 
(i) (#,) satisfies a large d«viation princwle with pow«rs (t,). 
(ii) There is a map 7 : ç--' [0, 1] such that 
(a) limsup#t°(F) N /(G) N liminfpto(G) for all F  .. G  çwth 
FcG. 
(b) "y(Uiel ai) = supiel "/(ai) for all {ai : i  I} C ç 
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(iii) There s et fiction f : .V -- [0, 1] such that 
limsupptâ(F) _< sup f(:r) _< liminfltâ (G) 
for «dl F G.. G E Ç u,ith F c G. 
(iv) There is a. fu.7,ction f: X  [0.1] stch that 
limsul,/,tâ (F) _< SUl, f(.r) _< lira inf p°(G) 

Jbr a, ll F G., G G ç with F c G. 
I.f (i) hot&. then (i) holds wth r«te fmctio .1 given 
(2.3) e-J= V{ f G [0.1] "x "f s«tisfi«s (iii) (resp. (iv))}. 
altd 
(2.4) 7¢-a = V{G [0.1] ç "7 satisfies (ii)}. 
q (il) holds u,ith , thon (i) hold.s" with rate f«nction - log G where 
J'v(a') = inf 7(G) for ail z  A. 
(f (iii) (rep. (iv)) bol& with f. thett (i) bol& u,ith rate fu,ctio -logf. 
Proof. If (il) holds with 7- then f. is upper semi-continuous and (i) holds with rate 
fim«tion - log f. by Lemma 2.1. 
If (iii) holds with f, then (il) holds with 7, and so (i) holds with rate function 
- log , since fw = f by Lemma 2.1. 
If (iv) holds with f, then pu 7(G) = suprcG supr f for ail G G ç. and notice 
that 7 satisfies (il). Thus (i) holds with rate fimction -logfT. Since f 
we hRvo 
sup f  sup f  sup f = sup f 
F F G G G 
for ail F  U, G G Ç with F C G (the equality follows from Lemma 2.1 (iii)). 
Thus (i) holds with ra.te fimcion - log f. 
If (i) holds, then (il), (iii) and (it,) hold. The fimçtion 
t, = {f  [0.1] x -f satisfies (iii) (resp. (it,))} 
obviouslv satisfies (iii) (resp. (it,)); the saine for h bv the preceding discussion, and 
h =  by the definition of h; put e -J =  and obtain (2.3). The map 
[0, 1] 9  7 satisfies (ii)} satisfies (il), and so (i) holds with rate funct.ion J given bv 
c-J = fTt. Since 7-a = 7t = 7t, (2.4) holds. 
Corollary 2.4. (Contractio principle) Let Y be 
t a coti.uous fu.ctio.. If (p) satisfies a large deviation priciple with powers 
(G) ad rate fueto, jx. the. ([p]) satisfies a large deviatio, pri.ciple with 
0 
powers (G) and rate functio, jv = 1 where l(v) 
Proof. Let jx be a rate fimction for (it ). The relations 
i.n .p [,d  (c) = in p G  (- I(F)) 
_j _d x 
 sup e  sup «  liminf[p]t(G) 
-(F) -(G) 
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for all F E .', G E çwith F C G and Proposition 2.3 show that (Tr[tt,]) satisfies 
a large deviation principle with powers (t) and rate functiol JY = -log] where 
jx jx 
f(/) = Sllprr-,(y ) e- for all g G Y (Sillce Sll])r-l(F ) e- = SllPF f). Equivalently, 
jr = l where l(g) = illfr-l(y ) .J\" for ail g G I'. [] 

3. A GENERAL VARIATIONAL FORMULA 
Up t,o now, the onlv known condition that ensures the existence of A(h) for all 
h  Ch(X) (and lnore generally for all [-c, +oe[-valued COllt.inllolls fllnctions ]2 on 
X satisfving the tail condition 
(3.1) lira limstlp/tW (e h/t l{eh>^t} ) = 0) 
is the existence of a large deviation principle with tight rate fimction, and A(h) is 
expressed in a variational form in terres of this rate fimction (Varadhan's theoreln, 
[2] Theorem 4.3.1). In this section, we generalize these results in two directions. 
First, Theoreln 3.1 gives the general variational form of lilnsllPltâ ' (e h/t° lr) for 
any Bord set }" C X and anv Borel measurable flln('tion ]/ on ." satisf.ving the 
localized tail condition (3.2). Next, Theorem a.a gives a sufficient condition for the 
existence of A(-) on Ch(X) which is also necessarv when X is nornlal; moreover, 
the variational form of A(-) is obtained in terres of any set-function 7 6 [0.1] 
satisfying the typical in-between inequalities of large deviations (a.ua). As a COl- 
sequence, Varadhan's theorem is gelmralized in various ways (Corollary 3.2 and 
Corollary 3.4). 
For each nlap h  X + [-oo, q-oo] we put F h - {e h E [ - 6,  q- 6]} and 
G h = {e h ]A - e, A + e[} for ail A > 0 and for all e > 0. 
Theorem a.1. For each Bord set t" C X. and for each [-0% +oo]-val.ued Bord 
measurable function h on X satisfgin9 
lira liln sup/zta  (e h/t° l{>^t}c)') = 0. 
hI--ec 

(3.2) 
we bave 
(3.3) 
(3.4) 

for some 11 e [0, +ce[. Moreover. li,nl, t(e h/t° 1,-) ezists if 

sup {(A-g) limilfpâ°(FÂ, efY)}= sup {(-e) lilnsupHta(Fîeç}')}. 
A_>0,e>0 A>_0,e>0 
Pro@ Let Y be any Bord subset of X. Put 9 = e, G.e = Gî,«Y, F.« = Fî.eY 
for ail A > 0 and for all « > 0.  have 
limsup«(91/tly)  limstlp(91/tlF.) 
fo If x 
 ( - e) lira sup g [ ,«) 
for all  OE 0 and for all e > 0, and so 
(3.5) limsupg(g/tly)  sup {(-e) limsup(F,«)}. 
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(3.7) 
Since 

Thus, in oder to prove (3.3) and (3.4), we have to prove that for SOlne/[ < oe, 
(3.6) lilns,,PPâ (gl/t«l,) i sup 
{x;«>o:g(x)_<M} 
For ail M  0, for ail N  N* and for ail 1 G j G N, we define 
Iç hae 
N 
1/t 
j=l 
 llI3X lims,,p l,  (glt 1FM..,) V liIllSllp « (gl/G l{g>M}y)" 

lira slip p (1/t ] FM,N,J ) (__ |illl 8/lp ]lta c' (&[,N,j)llg F,.. II, 
it follows from (3.7) that 
lira sup , (9 /t 1 ) 
(3.8)  max ]]91FM.]llimsuplz(F,,N.y) vlimsupl*(91/tl{e>t}m'). 
IjN ' 
Let AI  , N   in (3.8) and use (3.2) to obtain solne Alo  [0, [ such that 
(3.9) lilnsuplt«(gl/t«ly)  liminf lnax{I]glFM.m, lllilnsupp (F,NO)}. 
Thus, to obtain (3.6) it suffices to show 
lilll inf lnax { [91F,.. [1 linl 8up , (Mo,N.j) } 
(3.10)  8Up {(g(x)--)liln8tlp,oE(g(x),¢)}. 
If (3.10) does not hold, then there exists oe > 0 such that 
lira inf lnax { I[gl F»,,, H liln slip ,â (Mo,N,j) } 
Noe N- 
(3.11) > sup {(g(x) + p E) lilll sup t« 
- , 
Take 0 < o < oe/2 in (3.11) and obtain 
linl inf max {[]glFMo.m l linl slip Ira « (FMo.Nd) } 
(3.12) > sup {(g(x) + «o) lira sup " (G(.),«)}. 
But for ail 0  A  Mo and for ail N > Mo/«o we hae 
(3.13) (A + Eo) liInsup t l t'F 
, (a.)> lbu,., llil,p. t ,..) 
where jx is such that k e [(jx-1)ao/N, jxMo/N] (since [(jx-)Mo/', ix alo/N] c 
]A - «o, k + «oD- When A ranges over [0, klo], jx ranges over {j" 1  j  N}, and 
(3.13) implies 
O<<Mo 
(3.4) 2 max {llg,,..[limsup(F, NO)} 
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for ail N > Mo/eo. Notice that ri)r ail N E N* a.nd for ail 1 _< j _< N, if 
then j = Jg() for some x  Y. Thus it suffices to consider A  {g(x)  
Mo} in the L.H.S. of (3.14), that is, 
snp {(g(x) + eo)lin, supltL«(G9(),eo)} 
for all N > Mo/eo, which contradicts (3.12); it follows that (3.10), (3.6), and finally 
(3.3) and (3.4) hold. In the samc way that we obtainod (3.5), we have 
(3.15) liminf#L«(gi/t«ly)  su I) 
k0,e>0 
and the last assertion follows from (3.4). 
A localized version of Varadhan's theorem states that if X is regnlar an(l if 
satisfies a large deviation principle with t)owcrs (te,) and tight rate function .]. then 
(3.18) and (3.19) hold with 1 = J ([2], Exercise 4.3.11). Thc following corollary 
removes all the hypotheses on / and X. 
Corollary 3.2. Let 1 be a [0, +oe]-valued function on X satisfying 
(3.16) F  , limsup#L«(F)  supe -() 
(3.7) (re.»p. G  Ç, liIninfpL«(G) k suI) e-()). 
xG 
Tben, for each [-oe, +oe]-valu«d continaous function h on X satisfying (3.1), we 
(3.18) F  , liInsupltL«(«n/t"lF) 5 sup eh()« 
(3.19) (r«sp. G  Ç, lira inf ltL « («h/t 1) k sup «h()«-()). 
Proof. Suppose that (3.16) holds and (3.18) does hot hold for some [-, 
valued continuous function  on X satisfying (3.1). Since 
for all F G U by Theorem 3.1, there exists Fo G U, xo G Fo with h(xo) < OE, a.nd 
eo > 0 such that 
(e (°) eo)limsup/tL" F  

sup eh(x)e -l(x). 
xEF, h(x)< 

By (3.16) we have 
(e h(*°) - 0) sup e -l(*) > 
xE Fh( o) .o nF 
and so there exists x  Fh(o),o ç) F such that 
(e h(x°) -- ¢Eo)e -t() > sup eh(x)e-l(x. 

Since e h(*) _> e h(*°) - eo we obtain 

sup eh(*)e -t(x), 
xF,h(x)<oe 

xF,h(x)< c 

eh(**)e-l(**) > 

sup eh(*) e -t(x) 
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with xl Ç F and h(Xl) < , whence the contradiction. Supt)ose now that (3.17) 
holds and (3.19) does hot hold for some [-, +]-valued (.oltillUOUS ftulctioi1 h 
on X satisf'ing (3.1). By (3.15), there exists C0 Ç Ç such that 
sup e()e -/() > lira inf « (e /t« 1o) 
xEGo,h(x)<oe 
 sup {(e h(z) - ) lira inf p.â  (G he(«  Go)}. 
{x6Go 
Thus. there exists .vo Ç Go with h(xo) < , and u > {} such that 
ch(*°)e -t(z°) > u + sup {(e h(x) - ¢) lira inf p.  «"¢),e  Go)} 
{xGo,e>O:h(x)<oe} 

and 1)y (3.17) 

«t(e°)e-I(x°) > e t(x°) sup e -l(x), 
x- Gêh(xO) ,eO CIG 
which gives the contradiction.  
A direct consequence of Corollary 3.2 is that Varadhan's theorem can be stated 
verbatim for a general state space and with any function (in place of a tight rate 
flmction) I  X  [0. +] satisfying the large deviations lower and upper bounds: 
(3.21) limsupp,(F)  supe -() 5 supe -()  liminflt(G) 
xGF xGG 
for ail F Ç . G G Ç wi/h F C G: that is to say. A(h) exists and 
,X() = ,,p {() - («)} 
xX,h(x)<oe 
for ail [-OE,, +.]-valued continuous flmctions h on X satis6"ing (3.1). X will see 
wi/h Corollary 3.4 that it is possible to go fur/her in the generalization of Varadhan's 
theorem ot)taining/he saille coilclusiOlS with hypothesis weaker than (3.21). 
ecall that X is norlnal if and only if the following interpolation property holds: 
if f and g are real-valued respectively upper and lower seini-coiltinuous functions on 
.k" such tha/f  g, then there is a continuous fllllCtiOll  Oll X satis6"ing f 5 h  g. 
Theorem 3.3. Consider the followig statements: 
(i) A(h) exists for all h Ç Cb(X); 
(il) limsuppâ(F)  liminf p.â(G) for all F G . G G Ç with F C G. 
If X is normal. then (i)  (il). If (ii) holds, then (i) holds and moreover for each 
[-, +]-valued cont.inuous Iunct.ion  on X satisfying (a.1) we hat, e Ior some 

ai  [0, +[. 
(3.22) e h(h) = slip {()-e)7(Fî, e) } =- 
A>_0,e>0 
for all maps 7 " .7 tO ç -- [0.1] satisfging 
(3.23) 

{xX,e>O:eh() <_M} 

limsuppto(F) <_ -)(F) <_ 7(G) <_ lira inf 
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for all F E., G E ç with F c G. 

Pro@ Suppose that (i) holds and X is normal. For each F ff  and G ff Ç with 
F C G, there exists h  Cb(.') such that 1F  h  1 G. Since 1F  e nh-n  
e-"l.x.kG V 1G for all   N, we obtain 
lira sup p (F) 5 inf e A[nh-n) 
 inf liminf{e - + #tg(G)}  liminf p,(G) 
and (il) holds. 
Suppose that (il) holds. Let h be a [-, +]-valued continuous fimction on X 
satisfying (3.1), and 7 " lU Ç + [0, 1] satis(ying (3.23). Put 9 = eh and let us use 
the saine notation as in the proof of Theorem 3.1 (with l = A-). For all A  0. for 
all e > 0 and for all 6 > 0 with  > e, we have by (3.23), 
lira inf item(9 /«)  lira inf 1,(9 /t 
 (- 6)7(G,6)  (A- 6)7(F,«). 

Thus 

and 

lira iaf p.to (91/to) >_ lira (,k - 5)3'(FA,e) 
_> (,x- e)7(F,) _> ()- 

liminf p, tâ(91/t') >_ sup 
A>_O,e>O 
> ,,p 
A>O,e>O 
In order to prove (3.22), we bave to prove that for some M < 
(3.24) limsupp,(g 1Ce)  sup {(e h() - «)(G,«)}. 
{ x X,e >O:eh(    I } 
Bv using (3.23), and in the saine way that we have obtained (3.10) in the proof of 
Threm 3.1, we find some M0  [0. [ such that to prove (3.24) it suffices to prove 
which is achieved exactly as in Theorem 3.1. 

The following corollary gives sufficient conditions much weaker than large de- 
viations with tight rate function in order to have the conclusions of Varadlmn's 
theorem; in fact, we will see in the next section (Corollary 4.2) that when X is 
normal, the condition (3.25) is also necessary. 

Corollary 3.4. Let 1 be a [0, +oo]-valued function on X satisfying 
(3.25) lira sup p,to (F) < sup e -t(x) < lira inf lâ ° (G) 
x6G 
(3.26) (resp. lira sup #tâ (F) 
for alI F  . G E ç with F c G. Then, A(h) ezists and 
(3.27) A(h) = sup {(x) - 
xX.h(z)<oc 
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for all [-oc, +oc]-valued continuous functio. h on X satisfyi. (3.1). 
Pro@ Let h be a [-oc, +oc]-valued COl,tilmous fullction Oll X satisf.ving (3.1). If 
(3.25) holds, thon by Theoreln 3.3 (with 7(G) = supzec; e -t(z) for all G E Ç), A(h) 
exists and 
e A(h) ---- sup {(A -- e) sup e-l(z)}. 

Since for all A _> 0, e > 0 and x E Gî., 

(A -- e)e -l(z) <_ eh(z)e -l(z), 

we obtain 

and thns 
C A(h) _ Sllp eh(z)e -l(x). 
xEX,h(x)<oc 
For all x  X with h(x) < oc, and for all e > 0 we have 
(e h(x) -- ¢)e -l(z) <_ (e h(z) -- 6) sup 
YGêh() ,« 

which implies 

(eh(z) _ )e-l(:r) <_ e A(h), 
eh(z)e-l(z) <_ e A(h), 

and finally 
sup eh(z) e -(z)  e ri(h). 
xX.h(x)<oc 
Thus e h) = supex.h(z)< eh(X)e -t(z), which is equivalent to (3.27). If (3.26) 
holds, we conclude by applying Theorem 3.3 (with 3'(F) = supzeF e -(z) for ail 
F  9r), and replacing G h,.« bv. Fî.«, and G heh(), e bv. Fëh(), e in the above proof. [] 

Remark 3.5. Let F be the set of lnaps - : 9 r U Ç - [0.1] such that r(F) _< r(G) 
for all F  9 r, G  Ç with F C G. Define the narrow topology on F as the 
coarsest topology for which the lnappiugs 3' * 3'(}') are upper semi-contilmous for 
all Y E 9 r and lower semi-continuous for all Y  ç The net (#" (.1/t,)) can be seen 
as a net in F provided with the narrow topology, as well as a net iii [0, 
provided with the product topolog:y: Then. the implication (ii) => (i) in Theorem 
3.3 means that if (tttâ(.1/t«)) has a limit iii F, then (tttâ (-1/% )) has a limit in 
[0, oc[{h:heG(¥)}; lnoreover, the converse holds if X is normal. Of course, the 
limit in F when it exists is hot Ulfique: for each F  9 c and G  ç " defined bv 
-(F) = lira sup #tâ (F) and "(G) = lira inf ptâ (G) is an example, and "' defined bv 
")"(G) = 7(G) and 3/(F) = infa»F.aeç 7(G) is another oue. 
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4. CR1TER1A OF A LARGE DEV1AT1ON PR1NCIPLE 

In this section, we investigate what has to t)e added to the existence of A(h) for 
ail h E Ch(X) (in other words, of the limit A(-) of (logltt«(e /t° ))iii ]-o% +oc[ ch(X)) 
in order to have large deviations. Of course, SOnle hyl)otheses on X are required to 
have sufficieatly continuous functions; so we sui)pose here that X as normal. In this 
case, by Theorenl 3.3 (alad Relnark 3.5) the existence of A(-) on Ch(X) as equiva- 
lent to the existence of a narrow set-theoretic lilaait 7 E F of (pt«), which as also 
equivalent to tlae existence of A(h) for ail [-oc, +oc]-vahled continuous finl«tions 
h on X satisfying tlae rail condition (3.1); nloreover, fi)r eacla such finlction h. the 
variational form of A(h) as given ila terlns of '7. In I)articulal, "7 can vary and it as 
essentially this flexibilitv which allows us to o|)taila in Theoren 4.1 lmcessary and 
suflïcient conditions, each of thenl corresponding to sonle tyi)e of infornmtion: a 
property of A(-) as a fun«tional iii (ai), a special variational fol'ail of A(') in (iii), 
a property of "7 iii (iV), and a property of the net (pt) ila (v) and (ri). It as 
worth noticing that in hoth formulations (flmctional (il) or sct-theoretic (iv)), the 
condition on the lilnit as thc saine: a continuitv property on ilacreasing nets. As 
corollaries, several basic results of the thcorv are strelagthened bv realoving the 
tightness or compactncss tayi)othesis. 

Theore,n 4.1. If X as no,wtal, then the follou,ing statements are equivalent: 
(i) (p) satisfies a large deviation principle wilh powers (t,). 
(ai) A(h) exisls for all h  Cb(X), and A(b) converges fo A(h) for each increa.s- 
in9 net (ha) in Ch(X) converging poinlu,ise lo h  Cb(X). 
(iii) A(h) exisls for all h  Cb(X), and A(h) = supex{h(x ) -/(.r)} for some 
function ! : X  [0. +oc] and for ail h  Cb(X). 
(iv) There as a map "7 : Ç -- [0.1] such that 
(a) lilnsuplt(F) _< "7(G) < linlinfpt(G) for all F  ., G  ç wzth 
FcG. 
(b) "7([.J, Gi) = linl'7(G) for each increasing net (Gi) i Ç. 
(v) A(h) exists for all h  C(X), ad for all F  .T, for all open covers {Gi : 
i  I} of F and for all e > O. there exists a finte subset {Gq, .... G u } C 
{Gi : i  I} such that 

(4.1) lilninfpt(/)- limsuppt( U Gi) < ¢. 

(va) There as a function l : X -- [0, +oe] such that 
(4.2) 
inf/(x)= sup {-liminftclogtz(G)} = su v {-limsuptclogp(G)} 
xF GÇ,GDF GÇ,GDF 

for all F  .. 
If (i) holds with rate function J. then the following properties hold: 

(4.3) inf J(x)= sup {-A(h)} for all F  ,'; 
xGF h.Cb(X),hlF=O 
(4.4) inf J(x) = sup {-A(h)} for ail G  ç, 
xG hCb ( X ),eh _lG 
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where Che(X) is the set of [-oc, +oe[-valued bounded above cotiuous flmctions 
on X ; in particular, 

(4.5) J(x) = up {h(x)- A(h)} I« ll x e X- 

o 
(4.6) J = 1 yor ,,Il l" X -. [0. +] sti.yu,,9 (iii): 
(-1.7") e- J ( x ) = inf 
GEç,xEG 
]br all x e A. md for all 3" Ç  [0.1] sotisfying (iv): 
(4.8) ,l = 1 for all l" X  [0. +oe] satisfying (t,i). 
If morcover X is second cou,table, thon we tan replace "'net" bg "sequence" in (ii) 
Pro@ (i)  (iv) and (iii)  (ii) are «lear: (i)  (iii) bv Corollary 3.4 and so 
. V 
(i)  (ii) If (i) holds with rate flmction J. then for each F G . each open co er 
{Gi " i G I } of F and each e > 0, 

lira sul)l,â ° (F) < sup e -J < sul) e -J = sup sup e -J < sup lira inf pt« (Gi) ÷ 6, 

which implies (v). 

Suppose that (il) holds. We will prove that (i) holds. Let Che(X) be the set 
of I-et, +ec[-valued bounded al)ove continuous fimctions on X. By Theorem 3.3, 
A(h) exists in [-OE, +vc[ for ail h  Coe(X), and notice that 

A(h V k) = A(h) V A(k) 

for all k  Che(X); in particular. 

A(h V s) = A(h) V s 

for all s e [--ec,+ec[. Let (bi) be an increasing net in Che(X) converging to 
h  Che(X) with A(h) > -ec. For each real s < A(h) we have limA(hi V s) = 
A(h V s) = A(h), and so eventually A(h) > s, which shows that lira A(h) = A(h). 
Therefore, we can replace Ch(X) 1)y Che(X) in (ii). Let F e .T and h  Che(X) 
with hlF = 0. If A(h) > -oe, then A(hVs) = A(h) with (hVs)l F = 0 for ail 
s < A(h) A 0: if A(h) = -oe, then the sequence (A(h V --*))eN converges to --OE 
with (h V -n)l F = 0. Thus, 

inf e A(h) = inf e A(h), 
h Cba ( X ),h l oe=O h Cb ( X ),h l F =O 

and bv the interpolation property we have 

lira sup tt2 « (F) <_ inf e h(h) = inf e ri(h) 
h (7. Cba ( X ),h l F=O h (7.Cb ( X ),hl F =O 
(4.9) _< sup e h(h) <_ lira inf p (G) 
hGCba(X),eh_lG 

for ail F G .T, G G _G with F C G'. Define 

f() = 

inf c A(h) (= inf e't(h)) 
hGCb(A ),h(x)=O hGCb«(X),h(x)=O 
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for all x C A'. By (4.9), in order to prove (i) it sulïices to show that f is upper 
semi-continuous a.nd sa.tisfies 

(4.10) ,qU I) f(x) = inf e A(h) 
xE F hECb ( X ).h I oe =0 

for ail F C - and 

(4.11) su I) f(x) = sup e A(h) 
x:G hCb« ( X ),eh _lG 

for all G  Ç. We first show (4.10). Ch'arlv 

sui) f(x) <_ inf e A(h) 
xÇ F hCb( X),hlF=O 

for all F  -. SuI)I)ose that 
sup f(x) < e s < 

inf e A(h) 
hGCb( X ),hlF=O 

for some F  - and some real s. Then, for all .r  F there exists h« C Cb(X) «hich 
tan bc chosen negative such that ho(x) = 0 and 

(4.12) A(h«) < s < inf A(h). 
hECb(,\ ),hlF=0 

But 1F <_ e V¢ h with VeF h« bounded lower semi-continuous, and so there 
exists h  Ch(X) such that 1F N e h  e Ve h (in particular hIF = 0). Let I be 
the set of finite subsets of F ordered by inclusion, and bi = h A Vzei h« for all 
i  I, so that (bi)iCI is an increasing net in C(X) converging to h. Since A(hi)  
A(Ve h) = supzei A(hz) < s for ail i  I. we obtain limA(hi) = A(h)  s, which 
contradicts (4.12). Thus (4.10) holds.  now prove (4.11). Bv the interpolation 
property (between 1{«} and 1GE) ve have clearly 
sup f(x)  sup e A(h) 
xÇG hCba( X).ehlG 

for ail G  {ï. Suppose 

sup f(x) < sup e A(h) 
X _G h_Ctm (X) ,e h  1 G 

for some G  {ï. Then, for ail x  G there exists ha  Cb(X) with ha(x) = 0 such 
that 

(4.13) sup A(h) < s < A(ha) 
xG 

for some ha  Cba(X) with e ha <_ 1, and some real s. Let I be the set of filfite 
subsets of G ordered by inclusion, and bi = b A /ei hx for all i  I, so that 
(h)ie, is an increasing net in Cba(X) converging to hE. Then 

A(h) = limA(hi) <_ limA( V h)= lim(supA(h)) <_ s, 
xi xEi 

which contradicts (4.13). Thus (4.11) holds. It remains to show that f is upper 
semi-continuous. By (4.9), (4.10), (4.11), and since 
f(x) = inf e () _ < ilf sup e A(h) 
hÇCba(X),h(x)=O GD{x} hCb«(X),e  <_IG 
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for ail x E X. by Lennna 2.1 il suflïces to prove that 
(4.14) f(x) = illf e A(h) >_ |nf slip e A(h) 
hECba (X),h(e)=0 GD {e} hECba (X),e h 
for all :r E N. Suppose that (4.14) does not |iold for solne :r  X. Then. there 
exists h«  Cb,(X) wit.h hx(x) = 0, and oe > 0 such that 
(4.15) eA(h) + u < |nf sut) 
GD{x} hCb«(X},eh_lci 
I3y (4.9) and Theoreni 3.3 (with 7(G) = SllPheCba(.\-),eh<_lci eh(h)), we have 
(4.16) e A(h) = sui) {(A - ¢) slip eh(h)}. 
.X>O,e >O h_ Cba ( X ),e h (1 
Take A = 1 alid 0 < e < ,, in (4.16), and obtain by (4.15), 
mil» c A(h) < |nf sui» 
hCba ( X ),e h (1Gh x GD { e } hECba ( X ),e h  1 ci 
h, 
wit|l x  Gi., wh|ch gives the contradiction. Thus (4.14) holds and f is tlpper 
Selni-cont illllOllS. 
We have proved (i) = (ii), and that when (i) holds with rate flinction J, then 
(4.3) and (4.4) hold (by the miiqlleness of a rate flnlction on regular spaces); since 
A( - ()) = ,(,.) - (). 
(4.5) follows froln (4.3). 
Suppose that (iii) holds with l" X --, [0, +oc]. Then, obviouslv (il) and so (i) 
hold; let J be the associated rate filllctioll. By Corollary 3.4, 
(4.17") e A(h) --- slip ehe -J 

for ail h  Cb(X). and so 
(4.18) 

X 

Slip h--J = sup h-I 
X X 

for ail h  Ch(X). Clearly, for each h  Cb(X) there exists a real s such that 
Slip «he--J = sup ffhVse-J 

and 

and so by (4.18), 
(4.19) 

X X 

slip eh -l --- Sllp ¢hVse-l, 

X X 

SUp c e- J = Slip eh e -I. 
X X 
For any G  Ç choose an increasing net (h) in Cb(X) such that slip/e h` = 1, 
and obtain by (4.19), 
slip e - J = Slip e -I. 
G G 
o o 
Since supG e -I - supG e -I by Lemnla 2.1, we llave supGe -d = supGe -I for ail 
o O 
G  Ç. Since e -J and e - are upper selni-continuous, we have J = 1 by Lenmla 
O 
2.1. Thus, if (i) holds with rate fUllctiol ,J, then J = 1 for all 1  X 
satisf,ving (iii) and (4.6) holds. 
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Suppose that (iv) holds with 7 " {7 --, [0, 11. Define 7(F) = infa»F 7(G) for ail 
F E f', and notice that 7 is increasing on f', sat.isfies 7(F) <_ 7(G) for all F E f', 
G  Ç with F C G, and by (a), 

(4.20) -( [_J a)<_ S,lp-r(aj) 
I<j<_N I<_j<_N 

for each finite family {Gj}I<j<N 
h _ Che(X) and 
(4.21) e A(h) =- 

(4.22) 

C ç Bv Theorem (3.3), A(h) exists for all 

We will show t.hat (il) holds. Let. (h)e bc an increasing net in Ch(X) converging 
to h  Ch(X), and suppose that A(h) > supe A(h). By (4.21) and (4.22), there 
exists h0 > 0 and e0 > 0 such that 
(Ao - eo)7(G,« o) > sui) sui) {(£ - e)7(Fî,) } 
iI 
iI 

and thus 
h 
7EE(Gîo,« o) > sup 7EE(Fî,e o) > sup'y(GAo,eo). 
iI iEI 
Let » be the set of finite subsets of I ordered by inclusion, and obtain by (4.20), 
h 
(4.23) Vil e , ?(Co,«o) > sup?( ' 
G,«). 

But G h h 
ao,«o C sup¢e Uie¢G)o,eo, and the condition (b) contradicts (4.23). It 
follows that A(h) = supie/A(bi), that is, (ii) and so (i) hold: let J be the associated 
rate function. We now prove that (4.7) holds. Let G  ç and h  Che(X) be such 
that e h _< 1«. For all x  X and e > 0 with e h(x) > e, we bave 

and 

Thus, 
(4.24) 

sup 

Fêh(),« C G 

(eh(x) _ 

{(eh(:r)--g)'7(Fêh,=),e) } <_ sup 
{xX :,e h(=) >e},e>0 

<_ 

and since if e A(h) > O, 
(4.25) e A(h) = 

p {(eh() _ 
{xX;e h(x)>},>O 

by (4.24) and (4.25) we obtain 
(4.26) sup 
hCba(X),ehlG 

e ri(h) _< 7(G). 
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Sul)pose t hat 
sup e A(h) + ve < 
hECba( X ),e h lG 
for some u > 0. Bv taking A = 1 and 0 < e0 < u/2 in (4.22) we obtain 
(4.27) ?(Feo) + u/2 < ff(G) 
for all h  Cb.(X) such that e h  1G. Let (hi) be an increasing net in Cb.(X) such 
that sut) / e'h' = 1G, and let p be the set of finite subsets of I ordered 1)v inclusion 
bi 
Then (ieZ G,eo)e is an increasing net in Ç such that 

By (4.27) we bave 

and by (4.20), 

ri e ,, .,/(c,î:«o) + ./2 < ,,:(#* 
_ ,,«o) + ,/'2. < .,/(a), 

and by (4.4) 

sup e -a = -(G), 
G 
which gives (4.7) by upper senfi-continuity of e -d. 
Suppose that {v) holds. Bv Theorem 3.3. 
c A(h) .... sup {(A ¢)liminfpL"(Gî,e)} sui, {(A e) limsupp>(F,e) }h 
A0,e>0 AO,e>O 
for all h  Cb(X). Let (h)e« be an increasing net in Cb(X) converging to h  Cb(X). 
 will prove that limA(hi) = A(h). If A(h) > supie A(hi), then there exists 
Ao  0, o > 0 and oe > 0 such that 
(4.) 
(Ao eo)liminf h 
- (G,«o) > sup sup {(A-e)limsupp(Fî')}+u 
iÇI AO,e>O 
iEI AO,e>O 
iI 
Take eo < e < eo + u/2 in (4.29) and obtain 
(4.30) (Ao eo) liminf  Xo,«o/ > (Ao - eo)suplimsupt'Gh't o,«)" + /2. 
i6I 
Put F = F h Gi 
xo,eo' = Gxo,« for all i  I, and notice that F C ietGi" Since 
O 
FD G h 
xo.eo we obtain by (4.0), 
O 
lira inf 
iI 
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alld so 
liminfPâ(l) > lilnsut'#â( U Gij)+ t/2, 
for ail finite subsets {G., " 1  j  N} C {G " i e I}, which contradicts (4.1). 
Thus limA(hd = A(5), tiret is, (il) and so (i) hold. 
It remains to prove (i)  (ri), (4.8) and the last assertion. Supl)ose that (i) 
holds with rate function .1. Put f = e -a and let F  Y. By (4.3) we have 
sup f(z) = inf e A(h)  inf lintinfp'(G)  inf limsut)l,(G). 
x F hCb ( X ),hI F=O Gç,(IDF GÇ,GD F 
Suppose that 
inf e A(h) < inf liln sup tt  
hCb( X ),hlF=O GÇ,GDF 
Then. there exists  > 0 and hF  Ch(X with hl F = 0 su«h that 
e A(h) +  < inf liln sup p 
GÇ,GDF 

(4.31) 
Since 
(4.32) 

e A(hF) sup { (c hE(z) - 
by Theorem 3.3. by taking e = e0 <  iii (4.32) we obtain by (4.31), 
s t(F(, eo)} ) < inf limsupl,k (G). 
(4.33) sup {e h(x) liln up Pa 
{ x X ;h F( x) <  } , GÇ.GD F 
h 
Since hF(X) = 0 for all  G F, we have F C G.eo 
liln sup p ,hu  < lim sup p. 
,,« _ (Ft,«) < 

hF 
C F[,«o and (4.33) ilnplies 
inf lira sup pâ (G) 
GÇ,GDF 

By combilfing Theorem 4.1 with Corollary 3.4 and Lemma 2.1, we obtain in the 
following corollary necessary and suflïcient conditions in order that a large deviation 
o 
principle occurs with rate function the lower regularization 1 of a given flmction 
l" X --, [0, +oo]. Notice that by Proposition 2.3 and (i) ¢ (ii) in Çorollary 4.2. 
the infilnum of the set of [0, +oo]-valued functions I on X satisfying (3.25) coincides 
o 
with t.he lower regularization l of each its elements. The equivalence (i) = (iii) in 

and the contradiction. We have shown that 
sup f(x) = inf lira inf pta° (G) = inf lira sup pta (G) for all F  
xF GÇ,GDF GÇ,GDF 
which is equivalent to (4.2) with l = J. and so (ri) holds. 
If (ri) holds with l-X -- [0, +cci, then (4.2) ilnplies 
limsup #C(F) <_ supe - <_ liminf pâ° (G) 
F 
for all F  .T, G  Ç with F C G. By Proposition 2.3. (i) holds with rate ftlllCtiOll 
o 
l and (4.8) holds. 
If moreover X is second countable, then it is well knowll that for any familv 
{bi  i  I} of lower selni-continuous functions on X there exists a countable subset 
I0 C I such that supie bi = supie o bi. It is easy to sec iii the above proof that this 
property allows us to replace "net" by "sequence" in (il) (resp. (iv)), and "open 
covers" by "countable open covers" in (v). [] 
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Corollary 4.2 was known when 1 is a tight rate flmction ([2], Theorem 4.4.13): here 
there is no hypothesis on !. 
Corollary 4.2. Suppose that X is normal, and let I be a [0, + ]-valued function 
on X. Then. the following statements are equivalent: 
(i) (#a) satisfies a large deviation principle with powers (fa) and rate function 
o 
l. 

(ii) 

limsuppL " (F) _< sup e -(*) < lira lllft, a ((7) 
xEG 
for all F  , G  çwith F c G. 
(iii) A(b) exists and 
A(/,) = stlp{/,(x)- l(a')} for ail l, C Cb(X). 
xEX 
(iv) A(h) exists and 
A() = ,,p {h(«)-()} 
xX,h(x)< 
or all [-, +oe]-vol,,ed «o,tin,,ous unçtios  o X sati&ig (3.1). 
Pro«¢ (ii)  (iv)  (iii) by Corollary 3.4. (iii)  (i) by Theorem 4.1, and 
(i)  (ii) since supxeG e -(*) = sup,eG e -(*) for ail G  Ç by Lemnm 2.1.  
Recall that (Po) is said to be exponentially tight with respect to {ta) if for all 
g > 0 there is a compact set h C X such that lim supp (XkK) < «. Brvc's 
theorem ([2], Theorem 4.4.2) stores that if A(b) exists for all   Cb(X) and if 
(p) is exponentially tight with respect, to (t), then {p) satisfies a large deviation 
principle with powers (); moreover, the (necessarily tight) rate function satisfies 
(4.5). The following Corollary 4.a shows that the first conclusion holds under a 
hypothesis clearly weaker thon exponential tightness. Moreover, Theorem 4.1 stores 
that without anv tightness hypothesis, a rate function for (pâ) always satisfies 
(4.5). 
Corollary 4.3. Suppose t5at X is normal. If A(h) ests for all h  Ch(X), and 
gE for all open covers {Gi  i  I} of X. for all e > O. there exists a finite subset 
{«,, .... « } ç {«.  e } « a 
(4.34) limsupl,(Xk  Gi,) < g, 
t5e (I') satisfies a large deviation pri,,çiple with powers (). 
Pro4 Let F  , {G  i  I} be an open cover of F and e > 0. Then. 0e G O 
XkF is an open cover of X, and so there exists a finite subset {Çh,----Çi } C 
{Çi " i  I} such that 
linlsup#â(Xk(  Gi, UXk )) < g. 
IjN 

Thu8, 

lira sup #t (F \ U 
and bv Theorem 4.1, the conclusion holds. 
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Corollary 4.4. If X is 'normal and (lttâ ) satisfies (4.34), then (#,) bas a subnet 
(#) satisfyi.ng a large dew;ation principle with powers (t). 
Proof. Define A(h) = logp(e t/t) for ail h E Ch(X). Then. (A,(-)) is a net in 
the compact space [-et, +cci c(X) (with the product topology), and so there is a 
subnet (A(-)) converging to some limit A'(-). The result follows from Corollary 
4.3 applied to (/t ). [] 
Remark 4.5. (i) ** (iv) in Theorem 4.1 was known when 3' is a sup-preserving ca- 
pacity in the O'Brien sense, i.e., 7((7) 
Ç, (lç) = inf{7(G) : G D Iç.G Ç Ç} for all compact Iç c X, and -), satisfies (2.1) 
of Lemma 2.1 ([3]). Thus, Thcorem 4.1 removes the SUl)-preserving as well as the 
capacity conditions of-y (notice the difference between (iv) in Theorem 4.1 and (ii) 
in Proposition 2.3). In the spirit of Remark 3.5. this means that (It,) satisfies a large 
deviation principle with powers (te) if and only if (pâo) has a narrow set-theoretic 
limit in the set {7 G F : lim/(G) = ")(Ui Ci) for all increaing nets (Gi) in Ç}. 
Remark 4.6. When X is Polish and (p) 
Bell ([1], Theorem 2.1) implies the equivalen«e of the following statements: 
(i') (l) satisfies a large deviation prin«iple with powers (l/n) and tight rate 
fimction: 
(il') A(h) exists for all h Ç Ch(X), and infr A(hr) = A(h) for each decreasing 
sequence (bru) in Ch(X) converging to h  Ch(X). 
The equivalence (i) = (il) together with the last assertion in Theorem 4.1 tan be 
seen as a free tightness analogue of that, by replacing "decreasing" by 'increasing" 
in (ii'), and removing "tight'" in (i'). 
Remark 4.7. The relation (4.2) in Theorem 4.1 generalizes a well-known expression 
of a rate function J for (#tâ), obtained with 1 = J and F ranging over all singletons 
([2], Theorem 4.1.18). 
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INTEGRATION BY PARTS FORMULAS INVOLVING 
GENERALIZED FOURIER-FEYNMAN TRANSFORMS 
ON FUNCTION SPACE 

SEUNG JUN CHANG, JAE GIL ('ttOl, AND DAVID SKOUG 

ABSTRACT. In an upcoming paper. Chang and Skoug used a generalized Brow- 
nian motion process to define a generalized analytic Feynman integral and a 
generMized analytic Fourier-Feynman transform. In this paper we establish 
several integration by parts formulas involving generalized Feynman integrals, 
generalized Fourier-Feynlnan transforms, and t|le first variation of funclionals 
of the form F(x) ---- f((¢l,X),...,(¢n,X)) where (¢,x) don(,tes the t'aley- 
Wiener-Zygmund stochastic integral fo r c( t )dx( t ). 

1. INTRODICTION 

In [11], Park and Skoug, working in the sclting of one-parameter \Viener sl)a.ce 
C0[0. T] established several integra.ti(m t)y parts formulas involving analytic Feyn- 
man integrals, Fourier-Feynnmn transfonns, and the first variation of flmctionals 
of the form 

(1.1) 

where (.x) denotes the Paley-Wiener-Zygmund sto«hastic integral  
In this paper, we also studv flmctionals of the form (1.1) but with :r in a very 
general function spa«e Ço,b[O,T] rather than in the Wiener space Ç0[0. T]. The 
Wiener process used in [11] is flee of drifl and is stationary in rime while t.he 
stochastic process used in this pa.per is nonstationary in rime, is subject to a drifl 
a(t), and can be used to explain tlw position of the Ornstein-Uhlenbeck process 
in an external force field [10]. It turns out, as noted in temark 3.1 bclow, that 
including a drifl terre a (t) ma.kes est.ablishing various integration bv parts fonnulas 
for Fourier-Feynumn transforms very diflïcult. 
By choosing a(t) = 0 and b(t) = t oll [0, T], the function space C,b[O, T] rednces 
to the Wiener space C0[0, T], and so the results in [11] are i,nmediate corollaries of 
the results in this paper. For related work see [3], [4], and [6]. 
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2. DEFINITIONS AND PRELIMINARIES 
In this section we list the appropriate preliminaries and definitions ri'oto [5] that 
are needed to establish our parts forinulas in Sections 3, 4 and 5 below. 
Let D = [0, T] and let (t, B. P) be a probability measure space. A real-valued 
stochastic process Y on (-, B, P) and D is called a generalized Brownian motion 
process if l'(0, w) = 0 ahnost everywhere and for 0 = t0 < t < --- < t, _< T, the 
n-dimensional random vector (Y(tl, w),-.- , Y(t,,w)) is normally distributed with 
the density function 
Iç({:,,ï) = ((2r) ' H(b(tj)-b(tj_,))) -/2 
(2.1) 
{ 1- ((qj-a(tj))-('tlj-l-a(tj-1)))2} 
-exp - b(t)-b(t_) 
where -q = (q,-.-, q,,), q0 -- 0, ï= (t,.-.,t,), a(t) is an absolutely continuous 
real-valued function on [0,] with a(0) = 0, a'(t)  L[0, T], and b(t) is a strictly 
increasing, continuouslv diffcrentiable real-valued function with b(0) = 0 and b(t) > 
0 for each t G [0, T]. 
As explained in [13, pp. 18-20], Y induces a probability measure p on the measur- 
able space (]D 13 D where ]I Dis the space of all real-valued functions x(t), t G D, 
and D iS the smallest a-algebra of subsets of ]D with respect to which all the 
coordinate evaluation maps et(x) = x(t) defined on ]Dare measurable. The triple 
(]D BD p) is a probability measure space. This measure space is called the func- 
tion space induced by the generalized Brownian motion process Y deternfined by 
a(-) and b(-). 
Ve note that the generalized Brownian motion process Y determined by a(-) and 
b(-) is a Gaussian process with mean function a(t) and covariance function r(s, t) = 
min{b(s),b(t)}. By Thcorem 14.2, [13, p. 18ï], the probability measure p induced 
by Y, taking a separable version, is supported by C«,b[O, T] (which is equivalent 
to the Banach space of continuous functions x on [0, T] with x(0) = 0 under the 
sup norm). Hence (Ca.bI0, T], B(C«,b[0, T]), p) is the function space induced by 
where B(C«.b[0, T]) is the Borel a-algebra of C«,b[O, T]. 
A subset/3 of Ca,o[0, T] is said to be scale-invariam nmasurable [9] provided pB 
is B(C«,b[0, T])-measurable for all p > 0, and a scale-invariant measurable set N is 
said to be a scale-invariant null set provided p(pN) = 0 for all p > 0. A property 
that holds except on a scale-invariant null set is said to hold scale-invariant ahnost 
everywhere (s-a.e.). 
Let Lâ,b[O, T] be the Hilbert space of functions on [0, T] that are Lebesgue mea- 
surable and square integrable with respect to the Lebesgue-Stielt.jes measures on 
[0, T] induced by a(-) and b(-); i.e., 
L«,[O,T] = v" v2(s)db(s) <  and v2(s)dlal(s) < 
where lai(t) denotes the total variation of the function a on the interval [0, ri. 
2 
For u, v G L a,b[O, T], let 
T 
(2.3) (u, v)«,b = I u(t)v(t)d[b(t) + lai(t)]. 
J0 
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Then (-, .). is an inner product on L2,b[O. T] and I[ull,b = v/(u, u),b is a norm on 
L,b[O,T 1. In partioflar, note that Ilu[I«,b = 0 if and only if u(t)= 0 a.e. on [O,T]. 
N, rthermore, (L,b[O,T], . l],b)is a separable Hilbert space. 
Let {j}= be a complete orthogonal set of real-valued flmctions of bounded 
variation on [0, T] such that 
{0. 
(¢»¢«).b = . j = ., 
and for each v  Lâ,b[O. T], let 
(e.4) ,(t) = (,, ¢),.b¢(t) 
for  = 1,2,.-.. Then for each v  Lâ,b[O, T], the Paley-Wicner-Zygmund (PWZ) 
stochastic integral (v, x) is defined by the formula 
(.5) (,.) im ff 
= 
for all x  C«.b[O. T] for which the limit exists: one tan show that for each v 
c.d0.rl, the PWZ integral (v,x} exists for p-a.e, x 
g denote the function space integral of a B(Ca.b[O. T])-measurablc fimctional 
F by 
Jc 
whenever the integral exists. 
 are now ready to state the definition of the generalized analytic Feymnan 
integral. 
Definition 2.1. Let C denote thc complex numbers. Let C+ = {A  C  ReA > 0} 
and Ç+ = {A e C"  ¢ 0 and Rea  0}. Let F" C,b[O. T]  C be such that for 
each A > 0, the function space integral 
Jc 
..[0.TI 
exists for ail A > 0. If there exists a function J*(A) analytic in C+ such that 
J*(A) = J(A) for all A > 0, then J*(A) is defined to be the analytic fimction space 
integral of F over C.b[0, T] with parmncter A, and for   C+ we write 
(e.7) «' [1  _Z' [(x)] = *(a). 
Let q ¢ 0 be a real number and let F be a functional such that E  [F] exists for 
M1 A  C+. If the following limit exists, we call it the generalized analytic Feymnan 
integral of F with parmneter q and we write 
(.S) «'[1  «2"[()l = im 
--iq 
where A approaches -iq through lues in C+. 
Next (see [5], [7], [1], [8], and [6]) we state the definition of the generalized 
analytic Fourier-Feynnmn transform (GFFT). 
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Definition 2.2. For A  C+ and y  G,b[0, T], let 
(2.9) Ta(F)(9) = En'[F(9 + x)]. 
For p  (1.2], we define the Lp analvtic GFFT, Tq(p:F) of F, by the formula 
( e c+), 
(2.10 / Tqp:F)(9)= lin, T(F)(9) 
if it exists; i.e., for each p > 0. 
k" define the L analvtic GFFT. Tq(l: F) of F, bv the 

where 1/p + 1/p' = 1. 
fiwmula (A  C+) 

(2.11) Tq(l: F)(9) = lira Ta(F)(9) 
A--,-- iq 

if it exists. 

We note that flw l <_ p <_ '2. Tq(p: F) is only defined as s-a.e. \Ve also note that 
if Tq(p: F) exists and if F  G, then Tq(p: G) exists and Tq(p: G)  Tq(p: F). 
Next we give the definition of the first variation of a fimctional F on C«,b[O. T] 
tbllowed by a very fimdaln«ntal Cameron-Storvick type theorem [2] which was es- 
tablished in [5. Theorem 3.5]. 
Definition 2.3. Let F be a B(C«,b[O, T])-measurable fu,lctional on Ce.bi0, T] and 
let w  C«.b[O, T]. Then 
(2.2) aF(zl-a,) = F(.r + 
h=0 
(if it exists) is called the first variation of F. 
Throughout this paper, when working with 3F(z]w), we will alwavs require w 
to be an element of .4 where 
(2.13) A = {u, e C«,b[O,T] " w(t) = z(s)db(s) Ol- some z e L]b[O.T]}. 
Note that for F(z) of the form (1.1), aF(x[w) acts like a directional derivative 
in the direction of u,. For example, if f(ul,U2) = exp{3u + 4u2} and F(z) = 
((, ), (,.)), thl, 
6F(z[w) = [a(a, w) + 4(a2. w)] exp{a(a, z) + 4(a2.z)} 
= (c, )I, ((, ), (, ')) + (, ,,') f((» «), (, «)). 
The following notation is used throughout the paper: 

(2.14) 

alld 

(2.15) 

(u2,b ') = u2(t)b'(t)at = 

for tt  L2.b[O, T]. Furt.hermore for ail A  C+, v is 
real part. 

u2(t)db(t) 
alwavs chosen to bave positive 
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Theorem 2.1. Let z E L2,b[O, T] be given and fort  [0, TI, let w(t) = .1 z(s)db(s). 
For each p > O, let F(px) be p-integrable on Ce,bi0, T] and let F(px) hat,e a first 
variation 5F(px]pw) for ail x  C«,b[O, T] su«h thal for so'me positive fun«tio,t q(p). 
in t« ollo'wig euation «ist, t« t tird on« aIso «ists, d «qulit olds: 
(2.16) nf [SF(lw) ] - anf 
_ =-q [F(.)<z,«>] - (-@(=,a  [(x)]. 
In fact. for each A  C+, the above conclusions also hold for analgtic fwwtion space 
integrals, 
ç finish this section by stating a very fimdamental integration tbrmula fin the 
function stmce C«,b[O, T]. 
Let {1,"" ,«b} be an orthonormal set of funçti«ms from (L,b[O,], 
and for j  {1,--- ,zt} let 

T 
(2.18) 4j = (aj,a') =/o aj(t)da(t) 

and 
(2.19) Bj  (a,b')=  a(t)db(t). 
Note that Bi > 0 for each j ( { 1, 2,--. , z}, while for each j, A may be positive, 
negative or zero. 
Let f-   be Lebesgue mesurable, and le F(x) = f({al,X},.--, 
Then 
dc 
(2.20) Ç ,, )_} { (uJ-Aj)2}du, dan =  Y(.1,---..)p .... 
_ . ' =l 
in the sense that if either side exists, both sides exist and equality holds. 

3. INTEGRATION B'T PARTS FORMULAS ON FUNCTION SPACE 

Let n be a positive integer (fixed throughout this paper) and let {01,  -  , t't.} be 
an orthonorlllal set of ftlnctiolls fronl (Lâ,b[O, T], I1" lit,b)- Let m be a nommgative 
integer. Then for 1 < p < oe, let B(p; m) be the space of all functionals of the 
form (1.1) for s-a.e, x  C«,b[O,T] where all of the kth-ordel" partial derivatives 
fjl,...,j(Ul,''" ,Un) = fj,....,j(ff) of f: '-  are continuous and in LP( ") fOl" 
k ( {0,1,--. ,rn} and each j ( {1..--,n}. Also, let B(cx:m) be the space of all 
flnctionals of the forln (1.1) for s-a.e, x ( C«.b[0, T] where for k = 0.1,.-- ,m. 
all of the kth-order partial derivatives fil,... ,j» (ff) of f are in C0(), the spa«e of 
bounded continuous functions on  that Valfish at infinity. 
Out first lelnma follows directlv from the defilfitions of 5F(x]w) and B(p: m). 
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Lemlna 3.1. Let 1 <_ p < oc be given, let m bc a positive irteger, let F 6/(p: m) 
be given by equatio (1.1) and let w be an element of A. Then 
(3.1) F(xlw) = (oj,w)fj((Ol,£,... , 
j=l 
for s-a.e, x  Ca,[O, T]. Furthermore. as a function of x. 6F(-]w)  (p; m - 1). 
Lemma 3.2. Let p, m ad F be as ir Lemma 3.1. Let z  Lâ,[O. T] be 9iven. and 
for t e [0. T], let w(t) = fO z(s)db(s). Let G e B(p';m) be given by 
(3.2) G(«) = g((ol,x),-'-, 
o .-a.«.. z Ca.[0. T]. fi,, (') = F()G() o 
R  (1: m), azd as a flmction ofJ'. 5R(.Iw)  (1: m - 1). 
Proof. Let r(Ul,... ,Un) = f(ul,"" ,Un)9(Ul,''" ,Un). Then (x) 
--- , (a, x)) is an clement of (1: m) since all of the kth-order partial derivatives of 
r are contimous and in LJ() for k = 0.1,--- , m. Applying Lennna 3.1 we obtain 
that R(x[w), as a fimction of x, belongs to B(I: m - 1). 
Remark 3.1. Let F, G and R be as in Lemma 3.2 above. In evaluating 
z[n-x)], a,d E[5n-I,)] f  > 0. th 
(3.3) H(A;u,...,u)H(A:ff)=exp{-£(uJ-A)2} 
=1 
occurs, where Ai and Bj are given by equations (2.18) and (2.19) above. Clearly. 
for  > 0, H(A: ff)  1 for all ff G  since Bi > 0 for ail j = 1,-.-, n. But for 
 G Ç+, ]H(A;ff)] is hot necessarilv bomded by 1. Note that for each  G Ç+, 
 = c +id with c  ]d]  0. Hence, for each  G 
H(A;ff) =exp{ - (uj - Aj)29= 2Bj } 
= exp {  [(c2-d2+2«di)u-2(«+di)Ajuj+A] }- , 

and so 

(3.5) H(,: ff)] = exp { - - [(c2 - d2) -- 2-cAjuj + A'] } 
3 =1 -2Bi " 
Note that for , C C+, the case we consider throughout Section 3, Re() = 
c > ]d = hn()[  0. which implies that c 2 -d 2 > 0. Hence. for each A G C+, 
H(A; ff), as a function of ff, is an element of LV( ) for all p  [1. +]: in fact, 
H(A: ff) also belongs to C0(). These observations are critical to the development 
of the integration by parts formulas throughout Section 3. 
In Sections 4 and 5 below we consider the case where A = -iq  +-C+. In this 
ce,==«+idwith«==]d,. Hen«e, forA=-iq, qeN-{O}. 
c  - d 2 = O, and so 
(3.6) [H(-iq:ff), = exp{  [Ajuj - A] } 
=1 2B " 
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which is not necessarily ill LP(IR n) for any p E [1, +oc]. Thus, il Sections 4 and 5 
we will need to put additional restrictions Oll the functionals F and G in order to 
obtain the corresponding parts formulas iuvolving Fourier-Fevnmm transforms. 

Remark 3.2. Note that in the setting of [11], a(t) = 0 and b(t) = t on [0, T] and 
so Ai = (ctj,a') = 0 and Bi = (ct,b') = 1 for all j e {1,2,-.-,r}. Hence, for all 

IH(A:/7)I= exp - u =exp 

u <1. 
2 -- 
j=l 

Theorem 3.3. Let z e L,b[O.T ] be given and fort e [0. T], let w(t) = f z(s)db(s). 
Let p, m, F and G be as in Lemma 3.2. Then for all A  C+, 

(3.7) 

where x/ï is chose to have positive real part. 

Proof. First define R(x) = F(x)G(x) an, t let 

r(a,-.-,,) : f(u,.---.an)g(,q,"". ,). 
Then by Lemma 3.2, R E B(1; m) and (R(.[w) /3(1: m- 1). Furthermore ail of the 
kth-order partial derivatives of r are contimous and lu LI(iR n) for k = 0, 1.--. , m. 
Hence, R(px) is p-integrable on C,,b[O, T] for each p > 0. In addition, for s-a.e. 
x e C.b[O. T], 
5R(x[-w) = F(x)SG(xlw ) + 5F(x[w)G(x) 
= f((Ol,X),''" , (Otn,X)(Otj,W)j((OI,X),''" , 
(3.8) =1 
+ g((Ol,X),"" , (On,X)) '(Oj,w)fj((Ol,«),'", (an,x)). 
j=l 

But for ail u e L.b[O, T], 

(3.9) 

Jô0 T 

.()d,() 

u(s)z(s)db(s) 
T 
< fo Iu(s)z(s)ld[b(s) + lai(s)] 

In particular, since {eq,---,cn} are orthonormal, I(» ')1 < Ilzll.. for each 3 e 
{1,2.--. ,}. 



2932 SEUNG JUN CHANG, .IAE GIL CHOI, AND DAVID SKOUG 

Next, using (3.8) and (3.9), we see that for p > 0 and h > 0. 
<_ pllzll,»lf(<,,.p., + pi,,,,),..., (o.pa. + ph,v)) i 
Dut this implios that R(px + phwpw), as a fimction of x. is -integrable for ail 
p > 0 and h > O. This tan ho seon by intogrting the right-hand side of (3.10) terre 
by terre. For example, usin (2.20), ve see that for any ! E {1.--- ,}, 

(3.11) 

.exi,{_ - [uJ - P(Aj + h(aJ'w})]}du ...du n 
j= 2p2 B. 
Thus, using (3.10) and (3.11). we ol)tain that for p > 0 and h > 0. 
Next, using (3.8), (2.19). (3.3), and (3A), we see that for all A > 0, 
(3.12) =  [/()  (l. )l() 
+ () (,, )1,()] (: ). 
But, as noted in Remark 3.1 above, for each A  C+, H(A;/) is an element of 
C0(), and so the integrand on the right-hand side of (3.12) is in L(). Hence. 
eF [R(-I,)] = eF [F(')6(I) + F(I)()] 
exists for all   C+. A simil argument shows that the analytic function space 
integral E[F(x)G(x)] also exists for all   C+. Equation (3.7) nowfollows 
from Theorem 2.1 above; in particular, ri'oto equation (2.17) with F(x) replaced 
with R(x). 
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The following two corollm'ies are special cases of Theorem 3.3. 

Corollary 3.4. Let z, w, and m be as in Theotvm 3.3. Let F  B(2: m) be given 
by (1.1). Then for all A  C+, 
(3.13) AEn x  
= _ [(F(x)) (z,x)]- (:,a « [(F(x))/. 
Proof. In Theorem 3.3, «hoose p : 2 and Gx) 

Corollary 3.5. Let z and z2 bc elem«7,ts of L.b[O.T], and for t  [O.T], let 
wj(t) : fO zj(s)db(s) for j  {1.2}. Let F  (2:,n) be given by equation (1.1). 
Then ,for all A  C, 

(3.14) 

= AEaxn[F(a')SF(xlu,1)(z,x)]- V/--(z,a')En[F(a.)(F(X[Wl)]. 

Pro@ Let I' = 2 and G(x) = F(:rl., ) in Theorem 3.3. 

Lemlna 3.6. Let p. m and F be as in Lemma 3.1 obove. Then for all A  C, 

(3.15) 
for s-a.e. 9  C,b[O. T] "u,herc 

(3.16) 

,. = f(ff + OH(A: )dg[ 

with t?j and H given by eqaations (2.19) and (3.4) respectively. 

Pro@ For A > 0, equation (3.15) follmvs easilv fronl equation (2.20). But for 
each A  C+, as shown in Remark 3.1 above, H(A; u,--- ,u,) is an element of 
L()aC0(l ) for all p  [1. oe]. Hence, for each A  C+ and s-a.e, y  

f(//1 -[- (Ctl,/},---,/t n n c (Ctn,//))//():U1, --./An) 
belongs to L(NI ') and so equation (3.15) hohls throughout tE+. 

Our next lemma follows from standard results for convolution products. The 
key is that fol each A  C+, H(A: ff) is an element of L»(]K ') ç C0(]K n) for ail 
1 <_p<_ +oc,. 

Lemma 3.7. Let cho 
(a) If f  L  (]R ). 
(b) If f  LP(IR ) 
where p' = P 
p--l" 
(c) If f  

be given by equatioz (3.16) above. 
then ¢0(A; ")  C0(]R ') for all A  C+. 
for some p  (1, ac), then b0(A:-)  LP'(IR ") for all A  C+ 

then bo(A;-)  LI( n) for all A  C+. 

Our next theorenl follows immediately frolll Lemnla 3.7. 

Theorem 3.8. Let 1 <_ p <_ 
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Theorem 3.9. Let 1 < p < oe and w  A be gwen. Let F  B(p; m ) be gzven by 
«quatzo, (1.1). Then for all A  C+ and s-a.e, y  C«b[O,T], 

(3.17) 

, W) fl(al -- /l, y), - - - ,1tri -- 

-H(A;al,'-- ,an)dul...dun 

= T(5f(-I,,))(v). 

which, as a function of g. is an element of B(p'; m- 1). 

Proof. The fact that 5T.x(F)(y]w) is an element of /3(p';m- 1) follows directly 
from Theorem 3.8 and Lemma 3.1. To establish equation (3.17) for A > 0. simply 
ca.lculate 5T(F)(y]w) using equation (3.15), and then calculate T)(5F(.Iw))(y) 
using equations (3.1) and (2.9). Finally. equation (3.17) holds throughout C+ by 
analytic continuation in A. [] 

In our next theorenl we obtain an integration by parts formula involving Tx(F) 
and T,(G). 
Theorem 3.10. Let p. m, z, w, F and G be as in Theorem 3.3. Then for all A  C+, 
(3.18) 
E" [T(F)(x)ST(C)(xlw) + 5T(F)(x]w)T(C)(x)] 
= AEï'[T,(F)(x)T(G)(x)(z,x)]- OE(z,a')E''[T,(F)(x)T,(G)(x)]. 
Proof. For x  Ca,b[O.T], let R(x) = T(F)(x)T(G)(x). Then by Theorem 3.8, 
T(F)  /3(p'; m) and T(G)  /3(p: m). Hence, bv Lemma 3.2, R belongs to 
(1; rn), and so by Lemma 3.1.5R(x[w), as a fimction of x, belongs to (l'm- 1). 
Thus, equation (3.18) follows from Theorem 3.3 with F and G replaced by T,(F) 
and T.,, (G) respectively. [] 

Theorem 3.11. Let m,z and w be as in Lernma 3.2. Let p  [1.2] and let F 
and G in B(p:m) be given by equations (1.1) and (3.2) r«sp«ctively. Then for ail 
AC+, 
E.'[F(x)A(C)(xlw) + F(xlw)T(C)(x)] 
(3.19) 
= t'[F(.)T()(.)(,.)]- OE( -"-',, 
z,. )r.. [F(x)T.(G)(x)]. 
Proof. Let R(x) = F(x)T(G)(x) for x  Ca.b[O,T]. Bv Theorem 3.8, T(G) is an 
element of/3(/;m) and hence bv Lemma 3.2, R belongs to /3(1; m). Hence, by 
Lemma 3.1, 5R(x]w), as a function of x, belongs to/3(1"m - 1). Thus, equation 
(3.19) follows from Theorem 3.3 with (7 replaced bv T,((7). [] 

Corollary 3.12. Let m, z, w,p and F be as in Theorem 3.11. Then for ail A  C+, 

(3.20) 

E'[F(x)5T(F)(xlw) + 5F(x[w)T.(F)(x)] 
= E'[:()T(F)(.)(z,.)]- OE(z..')F[F(.)T(F)()]. 

Pro@ Simply choose G = F in Theorem 3.11. [] 
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Corollary 3.13. 

Let m, z and w be as zn Lemma 3.2. Let F E /3(2;m) be given 

2 

by equation (1.1). Then for all A  C+, 
Eoe n: [TA(F)(x)6T(F)(x]w)] 
(3.21) A 
__-- __ Ean. 
2  [(TA(F)(x))2<z'x] 

Proof. Siinply choose p = 2 and G = F in Theoreln 3.10. 

4. PARTS FORMULAS INVOLVING Tq(l: F) AND Tq(l: G) 
In this section we obtain various integration by parts fornmlas involving the 
analvtic GFFTs Tq(1; F) and Tq(l: G). In view of equation (3.6) above, we clearly 
need to impose additional restrictions on the functionals F and G than were needed 
throughout Section 3. 
Fix q e - {0}. Then as A  -iq through values in C+, « = Re()  /2 
and Idl  /2 where d = hn(). 
Next using equations (3.3) through (3.6) we see that for all A G Ç+ with c = 
Re() < ((1 + 

]H(A;ff) =exp{- 

(4.1) G exp 

<_ exp 

In addition. 

(4.2) 

1 

For f  La(IR ') let 
(4.3) 9r(/)(O = (27r)- 
denote the Fourier transform of f. 
Theorem 4.1. 
(1.1) with 

(4.4) 

, f(ff)exp { i Z ujçj }d 
j=l 

Let q  IR - {0} be given. 

Let F  /3(1; m) be given by equation 

1 
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for all k ¢ {0, 1,--. ,m} and each j  {1,--- .o}. Furthermore, assume that 

belongs to Co(lI). Then 
(4.) ¢o(-«:4) --_ _ -- 

f((+ ff)H(-iq: )d6 

is an, element of Co(II). Furthermore, the L1 analgtic GFFT, Tq(1; F) exsts as 
an elemeot of/3(c: m) and for s-a.e, g  Ce,bi0. T] is givet by tbe formula 

(4.7) Tq(I: F)(y) = ¢0(-1q; <al, g),'-  , <cn, g)). 
Proof. By (4.1) and (4.4) we know that f(.)H(-iq: .)  L(]n). and so its Fourier 
transform, ,T(f(.)H(-iq:.))(O cxists and bclongs to C0(]). Furthermore, bv 
cquations (4.6) and (3.4) and thc fact that vfZiq = c+ di = V/2 + di, we obtain 

1 

(4 8) 

- exp 

J= 2Bi B " ' Bn " 

Bv assumption (4.5), it follows that d)0(-iq: 0 is an element of Co(n). 
Finally, by equations (2.11), (3.15), (3.16), (4.8) and the dominated convergence 
theorem (the use of which is justified by (4.2)), it follows that for s-a.e, g  
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Theorem 4.2. 
more. assume that for each 1  { 1, 2,-.- , n }, 

Let q  IR - {0} and F  B(I m) be as m Theorem 4.1. Further- 

{ } ( ) 
q'l q n 
(4.10) exp - 9-[qAjJ (fl(.)H(--iq;.))- , ..... 

belongs to Co(IRn). Tken for eack 1  {1,2,---,n}. 

(4.11) çt(-iq:() = _  , ff((+ ff)H(-iq:ff)di[ 
is an element of Co(Rn). In addition, for each w  A and s-a.e. 
5Tq(I: F)(y]w) =  (t, w}Ot(-iq: (a,y), . - - , (a.,y)) 
(4.12) =l 
= (. aF(.I))(v), 
which, as a fuc*io of g. is a element of B(;m- 1). 

Pro@ The proof that each çbt(-iq; .) belongs to Co(IR ') is the saine as the proof in 
Theorem 4.1 above showing t hat ¢0 (-iq:')  Co (IR '). Equa.t ion (4.12) t hen follows 
immediately using the defilfition of the first variation and equa, tion (4.7). [] 

Our next theorem gives a parts formula involving F and Tq(I: G). 
Theorem 4.3. Let q   - {0} be given and let F  B(I: m) be as in Theorem 

4.1. Let G  B(I: m) be given by equation (3.2) with 

(4.13) 
for all k  {0, 1,--- ,m} 
(4.14) exp { - -Ç 

1 
2 =1 

and each ji  {1,--. ,ï}. Furthermore, assume that 

2X/-[Ajj } -(gl(.)H(_iq: .)) ( _ ql 
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belongs to Co(N n) for all l 
t e [O,T], let w(t)= f z(s)db(s). Then 

Lêt z  L,[O.T] be gwen and for 

E'fq[F(x)STq(1; G)(xlw ) + 5F(x[w)Tq(1; G)(x)] 
(4.15) = -iqEnfq[F(x)Tq(l: G)(x)(z,x)] 
- (-iq)(z.a')E'f[F(x)Tq(l'G)(x)]. 
Pro@ Let R(x) = F(x)Tq(l'G)(x). By Theorem 4.1. Tq(l:G)(x) is an element of 
B(; m) and so R(x) is an element of B(I: m). Also, by Theorem 4.1, Theorem 4.2 
and Lemma 3.2, 
6(x,,,) = F()6T(I: (1,) + 6F(,)T(: a)(x). 
as a fimction of x, is an element of B(I" m - 1). In addition, we know that for each 
I e {0.1,.-. ,.}, 
1/2 
Je[- 9(ff+ ff)H(-iq:ff)dff 

/'t(-iq:ç = _ 27rBj 
is an element of C0(IR n) with 

and 

for s-a.e. 
exist: 

(4.16) 

and 

Tq(l:G)(y) = bo(-iq: (c,y),-.-, 

6Tq( l: G)(y]w) = Z (ct, w)g't(-iq; (c,, y), . . . , (c,, y) ) 
/=1 
y E C,,b[O. T]. Hence, both of the following analytic Feynman integrals 

Exanfq[/(X)] = Enfq[F(x)Tq(l'a)(x)] 

= ( I @ ) /2 ,, f(ff)bo(-iq: ff)H(-iq', ff)dff 
j=l 

Exanfq [(!(X1732)] = Enf[F(x)STq(1; G)(xlw ) + F(xlw)Tq(l: G)(x)] 
- -(,-iq)/£[ 
(4.17) 
-- 
+ 0(-iq:ff) (at,w)ft(ff) H(-iq:ff)dff. 
/=I 
Also, proceeding as in the proof of Theorem 3.3 above, it is easy to show that for 
p>0andh>0, 
E[[6(px + pbwpw)] 
(4.18) _ 
+ 
/:1 
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Hence, by Theorem 2.1 above, the analytic Fe.wmmn integral 
E'G[R(x)(z,x)] = 
exists and equality (4.14) holds. [] 
Choosing G = F in Theorem 4.3 we get Ihe following integration by parts for- 
lnula. 
Corollary 4.4. Let q C OE- {O} be given and let F  B(1;m) be as i Theorem 
4. `) . Let z ad w be as in Theorem 4.3. Thez 
Eaznfq [F(:r)STq( l : F)(x[w) +  F(x[w)Tq(1; F)(.r)] 
(4.19) = -iqE"f[F(.r)Tq(l: F)(x)(z,x)] 
(-iq)½(z, ,,a,f 
- a lr, [F(.r)Tq(l:F)(z)]. 
Next we obtain a parts formula involving Tq(l F) and Tq(l: G). 
Theorem 4.5. Let q  OE- {0}. Let F  B(I: m) be as in Theom 4.2 and let G  
B(I: m) be as in Theorero 4.3. Furhermore, assume Hmt .for each l  {0.1,---  n}, 

(4.20) 1, .@(-iq: ff)H(-iq; ff) dg 
Then for w(t)= fô z(s)db(s) with z 

(4.2) 

Pro@ Let R(x) = Tq(l:F)(z)Tq(l:G)(x). Then R G B(oc,: m) and 6R(.rlw ), as a 
function of :r, is an element of B(oc;m- 1). Hence, by (4.6), (4.11) and (4.20), 
both of the following analytic Feymnan integrals exist: 

(4.22) 

and 

(4.23) 

1 
j=l 

b0(-iq: ff)ç'0(-iq; ff)H(-iq: ff)dg 

_ 2rrBj 

i 13. 
 qS°(-iq: g) E(at" w)@(-iq', g) 
n 1=1 

 ] 
+ bo(-iq; ff) E(oq, w)@(-iq;) H(-iq:)dff. 
/=1 

In addition, for p > 0 and h > 0, 
< pllzllo.» [],;bo(-iq; )ll,  II@(-iq; )11, 
(4.24) /=1 
+ II'b0(-/q; ")lloe E Ilçl(-iq; 
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Hence, bv Theorem 2.1, the analytic Feymnan integral E  [R(x)(z, )] exists and 
equality (4.21) holds. 
 finish this section with some examples which shed light upon the necessity 
of conditions such  (4.4) and (4.5), and which also illustrate that the conclusions 
of Lemma 3.7 are hot necessarily valid for A Ç Ç+ with e() = 0. 
In our first exainple ve define a flmctional F of the form (1.1) with n = 1, such 
that F is an elelnent of B(p; m) for ail p Ç [1, +], f is an element of L»(N) for 
all p  [1, +], and yet o(i; ") given by (4.6) is hot an element of Co(). In fact, 
[o(i:)1 = +OE for all ( Ç N. 
Exalnple 4.6. Let q = -1, let 't = 1, let mbe a nonnegative integer, and let 
be an elelnent of La»[0, T] with IIllla» = 1. Without loss of generality, we will 
smne that ,4 (see equation (2.18)) is positive. 
Let f " N + C be defined bv the fornmla 
{iuî iA,u, A AlUl } 
(4.25) I(,)  ç+Xo,+)()p 2e 2 + 2 4 " 
 note that 
2B1 4B " 
and hence f Ç LP(N) for all p Ç [1, +oe]. In fact, f is also an element of Co(N). 
ç then define F  C,[0. T]  C by the formula 
(.) F()  I((,, )). 
It is easv to see that F Ç B(p: m) for all p Ç [1, +]. 
Next, using equation (3.4) with  = 1, A = i, and  _ +i, we observe that 
(4.28) H(i; ul) = exp { Aul + iAu - A - iuî } 
2B " 
and hence 
(4.29) f(Ul)H(i:l)=U+'[o,+)(l)exp{ A } 
4B1 " 
which is not an element of Lp(N) for anv p 
Then, using equation (4.6) with n = 1 and q = -1, equation (4.25) and equation 
(4.28), we s that 
1 
() 
(i;) = 2B f(u +)H(i;ul)du 
1 
= 2ÇB exp  2B 
(4.30) - (u +)+k[o,+)(u +)exp iu + du 
B 4B 
1 
() {«î « 
=  exp 
2B 2B 
- (.  )+ ex  + . 
ç B 4B1 
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Thlls, 

(4.31) 
/_t-(x {tLtll v/Alttl } ] 
(Lt 1 ÷ 1) m+l exp -- + du.1 . 
 1 B1 -lB1 
Hcnce, choosing 1 - 0, and using the fact that Al is positive, we see that 
lao(i: O)l = (2B,)- u m+x exp 
: (2gB1) - 0 m+l exp  du = +, 
which implies that 0(i; ") is not an element of C0(R). hl fa«t. for each fixed   R, 
w observe that 
,0(/;,),=(2B)_exp { .4,1}4B1 
(4.32) 
(Ul+ exp + dul = 
and so 00(i; ") is hot an elelnent of LP(N) for any p  [1, +] even though f(.) was 
an elelnont of LP(N) for all p  [1. +] and F was an elclnent of B(p: m) for all 
p  [1, +oe]. Hence, the L1 analytic GFFT, T_(I: F) does hot exist. 
ç also note that f does hot satis6" condition (4.4) above since by equation 
(4.26) (recall that q =-1 and so ()/2 = 1), 

 lf(ul)]exp { IAlUll } dulB 

"tlUl } 
v"2.4ul + __ dal = +oc. 
4B1 BI 

In our lmXt example we exhibit a fUlctional F of the forln (1.1) that satisfies 
conditions (4.4) and (4.5) above. Furtherlnore, we are able to evaluate the integral 
in equation (4.6) and thus obtain a forlnula for ¢0(i: ) which does not involve anv 
integrals. 

Example 4.7. Let q = -1, let m be a nonImgative integer and let r be a posi- 
tive integer. Let {ai,---, a} be an orthonolnml set of functions froln (L.b[0. T], 
1" [«,b). and for ea.ch j e {1.---.n} let Ai and B be given by (2.18) a.nd (2.19) 
respectively.  define f : N   Cbv the forlnula 
[u - iAuj + A - u -- 
(4.33) f()  exp . 
J= 2B 

}Ve note that 
(4.34) 

j=l 2Bj 

and hellCe f e LP(] n) for ail p e [1, +o1. Also, f e Co(IR). 
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Let F " C,.b[O, T] --, C be given by 

(4.35) 

F(X)  f((l,x),"" ,(a.,x)). 

It is easy to show that F  B(p; m) for all p  [1. 
Next, using equation (4.33), together with equation (3.4) with A = i and x/ï = 
1+{ it follows that 

(4.36) 

f(ff)H(i; ff) = exp { - uj 

Now clearly f(.)H(i: .) is an element of LP(N ") 0 Co(N ") for ail p  [1. +oe]. Next, 
using equations (4.6), (3.4) and (4.33) we obtain 

(4.37) 

 f exp i Ujj 
1 j:l 

because 

(4.38) 

HClIC% 

(4.39) 

Ioo(i;()1 =exp- 2Bi ' 
j=l 

and so 00(i; ") is an element of C0(lI n) N LP(I n) for all p e [1, +oc]. 
2 
We also note that because of the factor exp{ 2--ï, } in the definition of f(ff) 
given by equation (4.33), condition (4.4) certainly holds. In addition, condition 
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= Bj 1¢0(i; 

Hence, by Theorem 4.1. the Lt 
g  

5. PARTS FORMULAS INVOLVING Tq(2;/) AND T(2:G) 

Note that in OllI" first theorem below we replace conditions (4.4) and (4.5) with 
condition (5.1). This condition is used to obtain a dominating flmction in order to 
apply the dominated convergence theorem. 

Theorem 5.1. Let q  IR - {0} be given. 
(1.1) with 

Let F  B(2: m) be given by equation 

for all k  {O. 1,... ,,n} and each j  {1.---,,z}. Then 
1 
(5.2) ¢o(-iq;& = _ 2rrB) ,, f((+ ff)H(-iq: g)dff 
is an element of L2(N). Furthermore. the L2 analytic GFFT. Tq(2: F) ests as 
an element of B(2ç m) and for s-a.e, y  C,b[O, T] is given by the formula 

(5.3) Tq2: F)(y) = ¢o(-iq; (c, y),. -- , (a,, y>). 
Proof. Using (4.1) we first note tllat 

If((+ ff)H(-iq: ff)ldff 

f((+ff)l exp {(1+}q1)2 

J= Bj 

1 
léo(-iq()l-< _ 2-7-D-7) 
1 
j= 2roB9 ] 

dff. 
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Hence. by (5.1) with k = 0. 

iàlld so 00(--iq;() is ill elen,t'nt of L2(lIr). 
To show that Tq(2: F) exists and is given by equation (5.3) it stlflîces to show 
that for ea«h p > 0. 

}ira /[ TX(pq) -cO(-iq; (o,.p./),'" , ((rt. py))12dl(l) 
-\--q. »[0,T] 

But 

No,v clearlv 4,0(A:)  00(-iq:) a.e. o,, IR  as A -- -iq through vah, es in C l . 
Tlms, to show that 

114,o(,x; .) - 4,o(-iq;-)112 ---' o, 

it suffices [11, p. 126] to show that 

114,0(,x: )112 --+ 114,0(-iq: )112 

asA -.iq through vah,es in C+. But forall A G C+ with Re(v/ï) < ((l+lql)/_)-. 

"9 
j=l 
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Hence, b¥ the dominated convergence theorem, 
lira 

0H(A: 

2 

5")dff 2 
f(ff + OIq(-iq; 

Corollary 5.2. Let q E IR - {0} and F  /3(2; m) be as in Theorem 5.1. Then for 
each !  { 1,2,..- , n}. 
1 
j=l 
is an element of Lu(N'). In addition, for ea«h w e A and s-a.e, g e Ce,bi0. T], 

(5.5) 

5Tq(2; F)(ylw ) = Z(a' w)¢,(-iq; (a,,g),-.- , 

= Tq(2:5F(.Iw))(y ), 

which, as a function of g, is an eleme'nt of/(2: m- 1). 

,)) 

Pro@ The proof that each ¢(-iq;-) belongs to L2(IR n) is the saine as the proof 
in Theorem 5.1 above showing that ¢0(-iq-')  Lu(IR'). Equation (5.5) then 
follows inmmdiately using the definition of the first variation and equations (5.2) 
and (5.3). [] 

Theorem 5.3. 
5.1. Furthermore, assume that 
(5.6)  [f ),,...,j (ff)H(-iq: ff)[dff < oo 
for all k e {0,1,---,nz} and «ach ji e {1,2,--. ,n}. 
Le G  8(2; m) be 9iven b equation (3.2) with 
1 
(5.7, £, [£ }gj,.....j«((+ff, lexp{ (1 ,q,)  
»r all k e {0,1,--. ,m} and «ach ji e {1,---,n}. 
4.3. Then 

(5.s) 

Let q  IR - {0} be given and let F  /3(2:m) be as in Theorem 

IAufil d d( < oc 

Let z and w be as in Theorem 
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Proof. By Theorem 5.1, for each I E {0, 1,--. ,n}, 
1 
(5.9) g,,(-iq;()= (fi)  
j=l 

t, gt(( + ff)H(-iq; ff)dff 

alld 

Exanfq [/(X)] = 27rBj 

Also, for p > 0 and h > 0, 
<_ pl[z[I«»( 1-I 2P2B) -½ IIf[12  ]]t(-iq;-)]12 
3=1 /=1 
+ ][o(-iq; ")112  liftIl2 
l-1 

/(ff)H (-iq: ff)Z/'0 (-iq: ff)da 

£ f(ff) 

+ "0o(-iq; g)Z(at, w)ft(ff) H(-iq:)d. 
l=1 

Hence, by Theorem 2.1, the analvtic Fevmnan integral E'q[R(x)(z,x)] exists and 
equality (5.8) holds. [] 
Next, choosing G = F in Theorem 5.3, we obtain the following integration by 
parts formula. 
Corollary 5.4. Let q. F  B(2: m), z, and w be as in Theorem 5.3. Then 
Enf[F(z)6Tq(2; F)(zlw) + 6F(zlw)Tq(2; F)(z)] 
= -qEnf[F(z)Tq(2; F)(x)(z,z)] 
- (-iq) (», ')z2e [F()T(: 

is ai1 elelnent of L 2 (11 ). Furt hernlore, 
(5.10) Tq(2: G)(x) = b0(-iq; (al.x),'." , (,x) ) 
is an clement of B(2; m), raid as a flmction of x, 
(5.11) 
/-----1 
belOllgS to (2;m- 1). Hel,Ce, /(«)= F(«)Tq(2:G)(x)is al, elenmnt of B(l:m) 
alld 
tiR(xlw) = F(z)6Tq(2: G)(.rlw) + 6F(xlw)Tq(2: G)(x) 
is ai1 eh?lllPnt of B(1; m,- 1). Since 
f(ff)H(-iq; ff)bt(-iq:ff) and ft(ff)H(-iq: ff)b0(-iq: 
belong to L(1R ) for eaçh / Ç {0, 1,-.- , u}, both of the following analvtic Fevmnan 
illtegrals exist: 
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Our final theorem is a counterpart to Theorem 4.5 above. 
Theorem 5.5. Let q C ]-{0} and let F and G be as in Theorem 5.3. Furthermore, 
assume that for each l  {0, 1,--. ,n}, 
./ ]¢,(g)(-q: )]d < . 
(5.12) 
Let z G L,b[O,T] be gwen and fort e [O,T] let w(t)= fô z(s)db(s). Then 
Efq[Tq(2: F)(.r)STq(2: G) (x[w) + 5Tq(2; F)(.rlW)Tq(2: G) (x)] 
-qE [Tq(2; F)(x)%(2:G)(x)(z,x)] 
(5.13) = - anf 
(--iq)  (Z, t,--anfq 
- « )x [Tq(2: F)(x)Tq(2:G)(x)]. 
ProoI. Let R(x) = Tq(2:F)(x)Tq(2:G)(x). Then R G B(1;m) and 6R(-[w) G 
E " (x 
B(1;m- 1). Using (5.2), (5.4), (5.9) and (5.12), we sec that [ )] and 
E"[6R(x]w)] both exist and are given bv equations (4.22) and (4,23) rcspec- 
tivclv. Finally, we sec that (5.13) follows from Thcorem 2.1, since for p > 0 and 

h>O, 

We finish t his paper wit h some very brief couunents about the functionals defiued 
in Examples 4.6 and 4.7 for the case p = 2. 
We first note that for the functional F(x)  B(2; m) defined by equation (4.27) 
with f(u)  L2(N) given by (4.25), the L2 analvtic GFFT, T_(2;F) does not 
exist because I¢0(i;)1 = +oe for each   N. In fact, the Lp analytic GFFT, 
T_(p; F) does not exist for anv p  [1, 2]. 
On the other hand, it is quite easy to see that condition (5.1) holds for the 
fun«tion f(ff) given by equation (4.33). Hen«e, for F(x) defined by equation (4.35), 
the L2 analytic GFFT, T_(2:F) exists as an element of B(2;m) and for s-a.e. 
y  C«,b[O,T] is given by the right-hand side of equation (4.41). In fact, for all 
p C [1,2], the Lp analytic GFFT, T_(p: F) exists as an element of B(p';m) and is 
given by the right-hand side of equation (4.41). 
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THERMODYNAMIC FORMALISM 
FOR COUNTABLE TO ONE MARKOV SYSTEMS 

MI('HIKO Yi:EI 

ABSTRACT. For countable to one transitive Markov systems we establish ther- 
modynamic formalism for non-H61der potentials in nonhyperbolic situations. 
We present a new method for the construction of conformal measures that 
satisfy the weak Gibbs property for potentials of weak bounded variation and 
show the existence of equilibrium states equivalent to the weak Gibbs measures. 
\'e see that certain periodic orbits cause a phase transition, non-Gibbsianness 
and force the decoEv of correlations to be slow. We apply our results to higher- 
dimensional maps with indiffcrenl periodic points. 

§0. INTRODUCTION 
Thermodvnainic for|nalism for hyperbolic systems was satisfactorily established 
with Bowen's program ([2]). The existence of geuerating finite l\Iarkov partitions 
and analysis of Ruelle-Perron-Frobenius operators associated to H61der potentials 
allow one to show the existence of torique equilibrimn states that satisfv the Gibbs 
property (in the sense of Bowen) and exponential deca,v of correlations. Also, the 
pressure fllnctions are analytic and there is no possibility of phase transition (non- 
uniqueness of equilibrium stat.es). Furthermore, the analyticity pïoblem is strongly 
relateà to nnlltifractal problems and the zero of Bowen's equation determines the 
Hausdorff dimension of linfit sers arising from certain iterated flmctional systems 
([4], [8], [12], [22]). On the other hand, phase transition, failure of the Gibbs prop- 
erty and slow decav of correlations can be observed for many COlnplex svstems 
which exhibit common phenomena in transition to turbulence (the so-called Iz- 
terrn{tte«y). In this paper we shall construct mathelnatical models which exhibit 
such phenomena and for this purpose we shall establish thermodvnanfic formal- 
ism for non-H61der potentials in nonhyperbolic situatious in the following sense: 
generating l\Iarkov partitions are countable partitions and dvnamical iustabilitv is 
sbecponential (subexponential decav of cylinder sizes). More specifically, for count- 
able to one transitive Markov systelns we shall construct conformal measures oe that 
are weak Gibbs Ineasures for potentials q of weak bonded variation (WBV)(see clef- 
initions in §1) and show the existence of equilibriuln states t/ for q equivalent to 
the weak Gibbs measures oe. The conformal lneasures oe associated to q play im- 
portant roles as reference measures from the physical point of view, and absolute 
continuitv of equilibrimn srates allovs one to describe statistical properties of ob- 
servable pheltoInena in the physical sense. In order to clarifv typical reasons for 
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plmse transition, non-Gibbsianness and slow decay of correlations, we introduce in 
§4 a notion of idifferet periodic point associated to potentials ¢ of VBV. Those 
periodic points cause failure of sunnnable variation for potentials ç and failure of 
bounded distortion of local 3acobians with respect to the weak Gibbs measure u for 
¢ (Proposition 4). Then a construction of weak Gibbs measures u for ¢ admitting 
indifferent periodic points implies subexponentlal mstablht in terres of cylin- 
der mesures (Proposition 3). Furthermore, a construction of equilibriunl states  
for such  equivalent to oe (Theorem 6, Lenmla 15) allows us to show both phase 
transition (Çorollary 2 Theorem 8) and non-Gibbsianless of equilibrimn states 
(Theorem 5). bi particular, our results are applicable to the following piecewise 
CLsmooth countable to one Markov lllap8 T defined on bomded regions X C u 
with indifferent periodic points (T%'0 = x0, det DT(.r0)l = 1) for which the p 
tentials - log  det DT] satisfv neither sunnnable variation nor bounded distortion 
so that previous results cannot be applicable. 
Exau,ple A. (Inhomoge'neous Diopha,tine approximations [13], [15], [16], [17], [19]. 
[20], [21]). Let X = {(x,.q)  N" 0  g  1,-g  .r < -g+l} and define 
T'XXby 
1 - , . 
-- +  ___ 
ZN). This map admiIs indifferent periodic points (1.0) and (-1.1) with period 
2, i.e., IdetDY(1,0)l = IdetDY(-1.1)l = 1 and is related to a Diophantine 
approxilnation problem of inhomogeneous linear class.  
complex continued fraction transforlnation T  ¥  X on the dimnond-shaped 
region X = {z = .ra +" - 1/2 G a.a- G 1/2}, where a = 1 +{, by 
T(z) = 1/z -[1/Zll. Here [z] denotes la, + 1/2]a + [2 + 1/21, where z is 
written in the form z = aa + ,[] = max{n  Z I  G x}(«  N) and 
[] = m{n  Z I n < z}(a  Z- N). This transformation bas an indifferent 
periodic orbit {1,-1} of period 2 and two indifferent fixed points at { and -i.  
ç recall previous works related to thennodynanfic fonnaIism for countable to 
one Markov systems. For countable Markov shifl.s, O. Sarig proved the existence of 
conforlnal measures and equilibrium states associated to locally H61der potentials 
defined in [10] and D. Fiebig, U. Fiebig and the author proved the existence of 
equilibriuln states for potentials satisfying bounded distortion (supnk Çn < OE in 
the definition of WBV) in [7]. Our main Theorems 4-8 do ilOt satis" these as- 
sunlptiollS and Examples A, B show that they cannot be treated by methods in 
[0] and in [7]. rthernlore, for higher-dimelSiolml sy8teillS that are not symbolic 
systems we mav bave crucial difficulties in verifying the positive recurrence con- 
dition impod on potentials in both [10] and [7]. The infinite iterated functional 
systems that Manldin and Urbanski studied in [8] correspond to the local inverses 
of piecewise confornml countable Bernoulli systems in out sense, and the method 
nsed in [8] severely relies on the Bernoulli property which fails to hold for Example 
B. Moreover, a H61der-type condition, imposed on potentials for the existence of 
con%rmal measures and %r establishing a variational principle, is ilOt satisfied bv 
the important potentials -log ] det DTI for both Examples A and B. 
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In order to prove our theorems, we first give iii §2 ail appropriate definition 
of topological pressure for countalle to one transitive Markov systems with finite 
range structure. Our definition coincides with the standm'd one by using peri- 
odic points under certaiu conditions (Lemma 6) which tan be easily verified for 
higher-dimensional exanlples in §8. We also associate the topological pressure to 
the spectral radius of the Perron-Frobenius-lRuelle operator (Theorem 3). The es- 
sential issue for constructing both weak Gibbs measures u for 05 and equilibrium 
states p. for 05 eqnivalent to u is to derive Schweiger's jump transformations T* over 
full cylinders (see the definition in §1) with respect to which a local exponential 
instability (05) and a local bou'nded distortio (06) for potentials ¢ are satisfied. 
Then showiug the existence of a zero of a generalized Bowen's equation (GBE) for 
derived potentials ç* associated to T* (Lemnm 7) allovs one to show the existence 
of conformal measures that are weak Gibbs measures for  (Theorem 4). Under a 
mild condition which cannot be covered bv previous works, we show the existence 
of a zero of (GBE) in §3 by using a product formula of zeta funct.ions (Proposition 
1), which shows a nice relation between zeta functions for the original systems and 
zeta functions for the jump transformalions. We also construct a-finite conformal 
measures via induced maps TA over a single full cvlinder .4 (Theorem 7) in §6 by 
using some idea that appeared in a previous work bv M. Denker and the author 
[5] in which no evidence of the existence of weak Gibbs confornml measures was 
given. We establish the existence of equilibrium states p for 05 of WBV equivalent 
to the weak Gibbs measure u for 05 via a junlp transformation (Theorem 6) in §5 
and via induced maps (Lemma 15) in §6. Theu we can immediately see that the 
appearance of indifferent periodic orbits associated to 05 implies a phase transi- 
tion, i.e., failure of the uniqueness of equilibrimn states (Corollary 2. Theorem 8). 
We should remark that our construction via induced maps shows the existence of 
(countably man3- ) lnutually singular equilibrum states. In §8 we apply our results 
to higher-dimensiolml piecewise C  Markov maps with indifferent periodic points. 
AI1 proofs of results in §§2-3 are postponed to the Appendix. 

§1. PRELIMINARIES 
Let. (X, d) be a compact metric space and let T  X -+ X be a noninvertible map 
that is hot necessarily continuous. Suppose that there exists a countable disjoint 
partition Q = {-¥i}iÇI of ,¥ such that iÇI intX is dense in X and the following 
properties are satisfied. 
(01) For each i  I with intXi ¢ O,T[itx, "inlXi  T(intXi) is a holneomor- 
phisln and (T]x,)- extends to a homeolnorphisln ri on cl(T(inlX)). 
(02) 
(03) {Xi}ieI generates , the a-algebra, of Borel subsets of X. 
We say that the triple (T, X. Q = {X}eI) is a piecewise C°-invertible system. 
By (01), F[tx, extends to a holneomorphisln (v) - on cl(intXi) for i e I 
with intXi ¢ O. For notational convenience we denote (v) - = Tcl(zntx). Let 
i = (i ... i)  I' satisfy int(X  T-1Xi ... T-(n-)Xi) ¢ O. Then we define 
X := Xii  T-1Xi=  ...  T-(n-1)Xi, which is called a cylinder of tank n and 
write Iii = - UN (01), TI,,x,1 .....  imX,1...i  + ç(int(X...,)) is a homeo- 
morphism and (Tn]intXil...i)-I extends to a homeonorphism Vil o vi= o ... o vi = 
Vil...i,"cl(T(int-k))  cl(intXt) and (Vil...in) -1 = rl,,,x...,.» x> impose 
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on (T, X, Q) the next condition, which gives a nice comltable states swnbolic dy- 
nallli£s Sil,lilar to SOfi£ shifts (cf. [15], [161, [171, [19]): 
(Finite Range Structuv).  = {it(T"Xh....) " VXh....,Vn > 0} consists of 
finitely many open sul)sets U1, .... UN of X.  
In particular, if (T, X. Q) satisfies the Markov property (i.e., inXi  intTAÇ.  
0 ilnplies inTXj D idX), then lg = {it(TX)  Vi  I} and we say that (T, X. Q) 
is an FRS Markov sys*«w.. If Ni  Q satisfies cl(T(i*Xi)) = X, then Xi is called 
a ,[vll clinder. If all cylinders arê full cvlinders so that  = {iLY}, then (T, X. Q) 
is called a Beruoulli sWtem. " assulne further the next transitive condition: 
N 
(Transitivitg). btX = U= ut and Vl E { 1,2 .... , N}, 30 < s < OE such that for 
each k E {1,2,..., OE'}, U contains an inlerior of a cylinder ,Y(t0(s) of rank s 
such that T ' (itX(k')(sz)) = U.  
The transitivity condition allows one to establish the next fact. 
Lenllna 1. Therc exists (} < 5' <  su«h that Ts(Uï= intX(k'O(s)) = intX and 
VXi  Q. TS+Z(idX,.) = intX. 
Pro4 4 L«mma 1. Since each Uk contains Uï=] intX(k'l)(,), choosing S =  st 
is enough t.o establish the desired fact.  
Re'mark (A). If (T, X. Q) is a Markov systeln, lhen lhe transitivity condition implies 
«periodicitg in the following sense: S > 0 such that VU, U  H,V > S.  = 
(iZ ...in) with itX ¢  satis{ving btXi, C Uk and T(int(Xi,)) = U. 
Definition. } say that  is a potential of weak bouuded variation WBV) if there 
exists a sequence of posit ive lmmbers { Cn } sat isfying lim  (1/ ) log C = 0 and 

Vr k 1,VXi,...i E V:o T-]Q, 
StlPz6xi .... 
illf«6xq .... 
Define 

temarl (t). 

n--i 
exp(Ej=0 ¢(T  x)) 

n-1 

If lar(T.b) - 0 as n -- ec,, which implies COlltilmity of ¢ iii 

symbolic distance, then çb satisfies the \VBV property. Hence if (T, X.Q) is a 
subshift of filfite type, then anv contilmous ftlllCtiOllS satis[v the WBV property and 
if (T. X. Q) is a countable Markov shift, then any unifornflv continuous functions 
 with l'oq(T, ¢) <  satisfv the WBV property ([7]). 
Let  be the «-algebra of Bord sers of the COlnpact space X. 
Definition ([17], [18], [20]). A proba.bility lneasure p on (X,) is called a weak 
Gibbs measure for a fllnction ¢ with a constant P if there exists a sequence {K }>0 
of positive numbers with lim+(1/n)logK = 0 mlch tha.t -a.e.,r, 
ç2' < (X'" 0"))  çn, 
where Xi...i. (a') denotes the cylinder containing m. 

l'a,'n(T, 0) := sup sup le(x) - 0(g)l- 
}--- ,n--1 
eVj=o T-J(Q) ac,yEY 
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For a fimçtiou b  X - OE, we define al opertor £ iy 
If  satisfies l'ar(¢) + 0 (, + ), II£lll := sup.x £4l(.r) <  and 
(04) {t'i}iG I i tll equi-cmlimous familv of partially defilwd mfiformlv «onlilut- 
O/lS 
then £ç preserves C(X) (i.e., £ " C(X) + C(X)) and is callcd thc Ruelle- 
Perrot-Frobenius operator. Ve rcmark that (04) is valid if 
V=o T-2(Q)} + 0 as . + 
 recall the uexl result, whi«h follows rioto Theorem 5. l in [17] and Propositiou 
. i [18]. 
Len,ma 2 ([17], [18]). Let (T.X.ç) be a t,'a,,.sitiee FRS Ma,'t'oe 
intX G . and let  be a potential qf IVBI'. Assume tlmt lh«re e.rist p > 0 and a 
Borel pmbabilit9 mea.sure u ot (X. U) sati.sf9inq £u = pp. whcre £* is the dual of 
. 
£. Thetz e i.s a weak Gibbs mca.ure .for  with -logp. 
Definition.  say lhat a Borel probalfility mcasure u on X is at f-conformal 
measuzv if d(uT)] , 
In order to show the weak Gibhs prol)erty of oe. we use thc [ollowing formula of 
thc local .]acobians with respect t() oe  
dx, 
Thus for the existence of weak Cil»bs measurcs, it is enough to show the existence 
of conformal measures (see 3). 
Lemma 3 (Theorem 2.2 iu [18]). Let u be a 'w«ak Gibbs measure for O u,«th -P. 
If there e:eist.s a T-ittvariant ergodic pmbabilit9meastte i t equivalent fo u with 
In particular, if the constant Pis the measme-theorctical pressure, thon the 
existence of a T-invariant ergodi« probability measure i t equivalcut to thc weak 
Gibbs measure u for ¢ with -P implies thc existence of a« cquilibrium state for 
ç (see 4). Iu order to achievc both coustructions of conformal measures and 
equilibrium states, we need to introduce new dcrivcd systems which are callcd jtmp 
transformatios ([13]). Let B C X be a uniou of cylinders of rank 1 of whk'h iudcx 
i belongs to a sui»set. J of I, and let Dt := Bï. Dcfinc a function R  X  NU {} 
by R(.) = inf{n  0  T G Ba } + 1 and for each « > 1. dcfine inductively 
= Y  T.- 
ç*3" TR()3 ". X denote X* :=, (U=o (N.>o{R(-r) > "})) and 
r := U {(,....) < r'- x,,.... ç ,,}. 
n>l 
Then it is easv to see that (T*, X*. Q* = {XL}6I. ) is an FRS Markov svstem. For 
= =o 5T (.r). 
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Definition. We say that an FRS Markov system satisfies local exponential insta- 
bility with respect to B if (05) " B 0 < 3'* < 1, 30 < F* < oc Sllch that Vn >_ 1. 
n--1 
er\", T'(n)=sup{diam}'l}'e V T*-'(Q*)} <_r*3' *n. 
j=0 
Definition. We say that a potential çb  X - IR satisfies local bounded distorsion 
with respect to B if there exists 0 > 0 such that ((16)  Vz_ = (iL ..  il*_l) E I*,  0 < 
Le(/) < oo satisfying 
Ici(ri(x))-d?(vi_(y))l <_ Le(i_)d(x,Y) ° (Vx.y e Tli-'Xi_) 

and 
sup Z L¢(ij+...ilgl) < 
/I* 3= 0 
Under the conditions (05-06), we can easily verify that {@*v    I*} is an 
equi-H6hlr contim,ous fmnily (cri [19], [20]) and % Var(T*,ç*) < . Both 
conditions (05-06) can be eilv verified for all higher-dimensional examples in 8. 
In the test of this section we shall state relations between jump transformations 
associat.ed to B and induced maps over B. Let Rs " B   U {} be the first 
return function defined bv Rs(a) = inf{n k 1  Tn G B}. Then we define the 
induced mapTB over {xG B " Rs(w) < oe} by Tsx = Tn()z and the induced 
Rs ()- Th(a)" Then 
potential B, " {x B  RB, (x < oe}  N by çBx (X) = 
we can immediately see the following facts. 

 +(0-¢oT,), 

Lemma 15 (Lemma 4.1 in [18]). Suppose that B consists #fall cylinders. Then for 
 -  isaTt3- 
any T-invariant probability measure m witb m(B) > O, mB .-- m(Bx) 
invariant probability measure and m* := mBT[[3 is a T*-im,ariant probability 
measure, m can be written in terres of m* by Schweiger's formula (see (3) in §5) 
and in terres of mt3 by Kac's formula (see Lemma 16). 

§2. TOPOLOGICAL PRESSURE FOR POTENTIALS OF ,VEAK BOUNDED VARIATION 
Let (T, X, Q) be a transitive FRS Markov system and let çb : X -- ]R be a 
potential of WBV. For each n > 0 and for each U  L/ we define the following 
partition functions : 

i:li_]=n,int(TX,, ):UDintX, x v£x=xcl[intX£) 

n--1 
exp[ Z (pTh(x)], 
h=0 

z,,(u, ¢) = 

n--1 
E sup exp[ Z dpTh(x)] 
i:[i_[=n,int(TX, )=UDintX, xX_ h=0 
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and 
z(u, ¢) = 
We further define 
z,, (,):= 

i_:li_l=n,int( T X ,, )=UDit X q 

inf exp[--" 6Th(x)]. 
xEXi_  
h=0 

n-1 
Z Z exp[Z 6Th(x)]" 
i_:li[=.int(TX,n)DintX,  vi_x=xEcl(intX_) h=0 
We shall define the topological pressure as thc asymptotic growth rates of these 
partition flmctions. 
Theorem 1 (%)pological pressm'e for I)otcntials of WBV). Let (T,X.Q) be a 
transitive FRS Markov system and let  be a potential qf Il'BI2 For each U G , 
lina L log(U, ), lim  log ,(l r, ), lim L log Z(U, ) exist and do 
hot depevd o r. Furtheore, the limits coincide with lim L logZ(). 
1 log Z,,() the topolo9ical pvssure for 
 call the limit Ptp(T. ) := lim,.oe  
. The next fa«t tan t)e verifie(l easilv. 
Lelnma 6. Uvder the next conditiom Z() coincides with the us'ad partition 
(1) For Zo  X,...i, with Txo = xo, either xo  intXi...i or .r0  clXj...j 
.for(j...j) ¢ (i...i). 
Let ç be thc filfite disjoint partition gcnerated by .  shouM claire that 
if a periodic poiut x0 with period , is COlltaincd in a cyliudcr Xi...i, satisviug 
Xi...i, C itl  for SOlne 1  G ç, then '0  OXi...i, If hot, we have a contradiction 
to Xo  intl  because of xo  T(OX...i,) = O(TnXq...i). By usiug this fact. we 
will see that all higher-dimensional exanlples in 8 satisfy (1). The Artin-Mazur- 
Ruelle zeta function @.e(z) is defined by @,e(z) = exp[, Z()]. Then the 
radius of convergeuce of Çr, e(z) is giveu by pe = exp[limsup  logZ()] -. 
ç define 
W(T) := {: X   I  satisfies WBV and Ptv(T,) < OE} 

and 

WB(T) := {6 e W(T) [ l'ar,6 --, 0 (,, --, 0),11611 := 
We can easily see that the pressure fulmtion Ptop(T, .) : W(T) --, 1I{ satisfies conti- 
lmity, convexity aud V6, 62 c W(T), Ptop(T, cb + 62) < Ptop(T. 6) + Ptop(T. 62)- 
Furtherrnore. by applying Theorem 2.4 in [7] we have the follov«ing fact. 
Theorem 2. î;B(T) is a Banach space and Ptop(T, .) : î;B(T) --  is a Lipschitz 
continuous convex function. 
Definition. If an FRS Markov system (T, X. Q) satisfies that VU c L/, BXi  Q 
such that Xi C U and T(intXi) = intX, then (T, X, Q) is called a strongly transitive 
FRS Markov system. 
Theorem 3 (Topological pressure and the spectral radius). Let (T,X,Q) be a 
strongly transitive FRS Markov system satisfying Lt f 12 ¢ . Let ç be a potential of 
weak bounded variation. Then VU  lg f3 l? and Vx  U, linloe 1 logL;lu(x) = 
Ptop(T, 6). Futhermore, lim,> 1_, log IIL;ël][ = lim_oe L 
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\Ve can easilv verify ail conditions iii Theorem 3 for exalnples in §8. 

§3. THE CONSTRUCTION OF WEAK GIBBS CONFORblAL blEASU1RES 
Let (T. X. Q) lin a transitive FRS Markov svsteln and let  G (T). Suppose 
that there exists a union of fldl cvlinders B(C X) with respect to which (T. X. Q) 
satisfies local eXpOlential ilstabi}ity alld  satisfies local bounded disçortion. For 
pR(g)--I T h (£) We define 
the derived potential *(,r) = h.=0 
(]..4)Çl*n:it('F * Y )Dit., v]...nX=XÇcl{intXL..4n ) 
Then l»y Theoreln 1. Smlmmble variati(ms of * allow OlW to show that 
1 
 i,ll - g Z,,(*) := Pp(Y*,*) e (- l- 
Theorem 4 (A constru('lioll of confornlal lneasures via jmnl) transformations). Let 
(T, X. Q) bc z hmtsilite FR+" M«u'kof .sy8tem ittd lel  G }V(T). Suppose that there 
«.rist. « utiot «t( full c,lliuder., B(C X) u,ith re.spect o u,hich (T,X.Q) satisfies 
local e.rpotetttiM in.l«bilil9 ami  .ç«tti,sfies loco,[ boutded dislortion. .4s'zlte further 
lhat ]{£»*-nmin{a.&p(T*,ç*)} 1 Il < . T»«,, lhere emi.st.s a Borel probabilil m«asure 
 on X suppovled o X* .s.o, li.sfyitg 
As we bave amtounced in 0. for constructing a xxeak Gibbs measure for  of 
XVBV. we shall considcr thc following generalized Bowen's equation: 
The existence of a zero of the equalion (GBE) %llows Kom the standard argmnent 
in the case when 0  Ptop(T*,¢*) <  because of contimfitv of the function 
should notice that the uniqueness of thc zero of (GBE) follows Kom the "'strictly'" 
decreasing property of the fmtction, s  Ptop(T*, *-.sR) in the standard situation. 
Here we have no evidence of it although the funclion is decreasing. If Ptop(T*, *) < 
0. then mder the assumption Ptop(T*, * - RPtop(T*. *)) <  we see that 
Ptop(Y*, O* -- Ptop(*, *)) k Ptop(T*, * - Aop(T*, *)) = 0 
and so we can reduce to the previous case. If Ptop(T*. ¢*) = . then we cannot use 
the st.andard argulnent. Now we corne to state the next kev lemma, which allows 
one to establish Theorem . 

Lelnma 7 {The existence of a zero of (GBE)). (i) /f 0 _< Ptop(T*, çb*) < oc,, then 
Ptop(T, ) > 0 a,,d s0 _> 0 satisfgi,9 Ptop(T*, * - s0R) = 0. 
(ii) ff Ptop(T*, çb*) < 0 o,td Ptop(T*, çb* -/ï'Ptop(T*, çb*)) < oe, then Ptop(T, çb - 
Ptop(T*, çb*)) _> 11 a,,d s0 _> 0 sa, tisfgi,,9 Ptop(T*, (çb - Ptop(T*, çb*))* - s0/ï' ) = 0. 
(iii) /f sup{s G IR: Ptop(T*,c)* - sR) = oc} = lnill{O. Ptop(T*,O*)}, then s0 _> 
l,lil,{(), Ptop(T*. d)*)} .sttc[t ll-at Ptop(T*. d)* - s0) = 0. 
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We recall the formal power serics (T,4(z) = exp[,= 1 -g- ,(¢)] which is called 
the Artil>Mazur-l:hmlle zeta fimction. The next product formula of zeta functions 
plays an important foie in proving Lenuna 7. 

Proposition 1 (cf. [17]). II'e tan write 

(2) 

çr,¢(e×p(-s)) = çr-,«-R(1) × 

Corollary 1. /.f s > Ptop(T, 0), then Ptop(T*, O* -- .s/-¢) <_ 0. 

Bv Theorelu 3, the assumption ]]£O*-Rmin{0,Ptop(T*,O*)}l]] < C implies either 
0 < Ptop(T*, *) < oc or Ptop(T*, * - r¢Ptop(T*, *)) < oo is satisfied. He,lce 
it follows froln Lemma 7 that Ss0 > nlin{0, Ptop(T*, *)} satisfying Ptop(ff*, * - 
a'0H) = 0 and II:«,olll < . Since Q* = {x,_}i¢** consists of full cylinders and 
sulnlnahility of variations ,__ Iar,(T*. * - s0/i') < oo is valid, we Call apply 
P. Walter's argmnent in [14] to show the existence (,f ail exp[soR - ¢*]-conforlnal 
measure with resl)ect to T*. 

Lelnma 8. There exisls a Bmvl probability mea.'nw oe on X solisfging £.* 
¢._soRlY : 
and v(intX*) = 1. 

In §9 we shall show the existence of an exp[s0- ¢]-conformal measure for thc zero 
s0 of (GBE) by using the conformal measure oe on X* and show s0 = Ptop( T, 
which implies uniqueness of the zero of (GBE). At the end of this section, we 
shall consider the case when I]£¢._tni{o.e,o(r.,¢.)}lll = oc. B.v Tlmorem 3. if 
* n 
Ptop(T , ¢*-&,]ï') = 0. then therc exists suflïciently large n such that ]] (¢-,o)* 11] = 

]]£(¢_,)11] < oc, where 
--1 

(0- So) := Z (- s°)*T* = 

n-1 

We shall introduce a new stopping time (depending on n > 1) defined Oll X* bv 

R,(x) := inf{k _> n I Xi...i(x)  

n--1 r--I 
V r*-Q*} = 2 

Then a new jump transformation S* detiued bv S*(x) := 7"R(X)(x) is equal to T *' 
and the next facts can be verified easily. 

Now we shall consider a two-parameter falnily of functions {(¢ - s) I (s, n)  
]R x N} and the equations Ptop(T *n, (¢ - 8)z) = 0. Applying Theorem 4 gives the 
next result. 

Proposition 2. Suppose that ail conditions in Theorem 3 are satisfied. If there 
exist .s0  ]R and no  N such that Vn _> n0, Ptop(T *n, (¢ -- S0)z) = 0. then there 
exists a Borel probabilitg mcasure oe o X supported on X* satisfyin9 -4-T[X, = 
exp[s0- ¢](Vi e I) ad p(UiFio.\i)= o. Furtherrnore. so = Ptop(T,¢). 
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§4. INDIFFERENT PERIODIC POINTS AND NON-G1BBSIANNESS 
Let (T, X, Q) be a transitive FRS Markov system and let  E kV(T). The next 
lemma follows from the definition of Ptop(T, ) directly. 
1 q--I 
Lemma 10. Ptop(T, ¢) >_  h=o dpTh(xo)( VxO  X, Tqxo = 
Definition..r0 is called a generalized indifferent periodic point with period q with 
1 q--1 
respect, to  if Ptop(T, ç) =  h=0 (pTh(x0)  If x0 is hot indifferent, then we calt 
 fo a generalized repelling periodic point. 

If a potential  of \VBV admits a generalized indifferent periodic point, then we 
can observe interesting statistical phenomena. More specifically, if there exists an 
exp[Ptop(T, )- ¢]-conformal measure u, then t he above definit ions can be described 
in terres of the local .]acobians with respect to u, that is, 

d(uT q) 

q--I 
Ix,...,q(o)(xo) = exp[qPtop(T, dp) - Z cpTh(x°)] = 1. 
h=0 

Then we have the following facts. 

Proposition 3. Let .r0 be a generalized indifferent periodic point with period q with 
respect to c G kV(T). Let u b« an exp[Ptop(T,b)- O]-conformal m«asure. Th«n 
(i) Vs  1. Ptop(T, s0) = sPtop(, 0) and Vs < 1. Ptop(. sO)  sPtop(T. ). 
(il) u(Xi...i, (x0)) decaBs subeonentiall9 fast. 

Proof. Bv Lemlna 10, we have Ptop(T, sdp) > s  Eï dpTi(xo). In particular, if x0 
is a generalized indifferent periodic point for b, then Ptop(T, sdp) >_ sPtop(T, dp). We 
recall that by Lemma 2 the conforlnat measure u is a weak Gibbs measure for b of 
WBV. Then we bave for s > 1. 

1 1 

where both C, and Kn satisfv tim,_oe -1 ]ogC, = 0 and lilnn--,ec_1 logK, = 0. 
Since lilnn--,oe _1]ogCnIÇn = 0, we bave Ptop(T. sdp) _< sPtop(V, çb) for s _> 1. (ii) 
fotlows frolll Proposit.ion 6.1 iii [21]. [] 

Let us recall that u was obtained by constructing a jump transformation in 
Theorem 4. Then we can associa.te the generalized indifferent periodic points to 
the lnarginal sets Ç>_0 D. 
Proposition 4. Let xo be a generalized indifferent periodic point with period q with 
respect to dp  kV(T). 
(i) (Failure of bounded distortion) 
Cnq(XO) :-- sup exp[-ïq-l )Th(x)] - oc 
nq--1 
C'Xil ..... q(XO' exp[Eh_0 (bTh(/)] 
monotonicall9 as n --* 
(il) xo  Ç),>0 D,,. 
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Pro@ Since C,(Xo) is the distortion of d(vT') 
d over cylinders Xil...i(xo), (i) follows 
from Lemma 6.1 in [211. S,,ppose xo ¢ 0ao D. Then by Sublem,na A (see 9) we 
bave .ro  X*. Since  Vr,(T*, ç*) < oe ilnplies that C,q(Xo) cannot increase 
monotonically, we have a contradiction to (i).  complete the proof.  
Remark (C). We claire that 0 D, can contain repelling periodic points. 
If we have a T-invariant probability mea.sure p equivalent to u via Kac's formula 
(Lemma 16) or Schweiger's formula (3) in 5, then the invariant densities d/du 
are typically mfl)ounded at indifferent periodic poinIs with respect to u (Lemma 
6.2 in [21]) so that we tan sec interesting phenomena from a statistical point of 
view ([19], [21]). For example, under the existence of a generalized indifferent 
periodic point ,r0 with respect to , Ihe raie of decay of correlation may be slower 
than u(X..., (x0)), which decays subexonentially fast by (il) in Proposition 3.  
referee [21] for fllrther details. On the other hand, the Dirae measure m supported 
on the generalized indifferent periodic orbit with respect o  satisfies Ptop(T, ) = 
h.(T) + f. çdm. Hen«e if we can esIablish a variaIional principle for the topological 
pressure and tan construct a T-invariant measm'e p equivalent to Ihe weak Gibbs 
lneasure u for  wiIh -Ptop(T,), theu by Lemlna 3 we sec immediately failure 
of uniqueness of equilibriuln states. Furtherlnore, by the definiIion of indifferency 
we can show faihlre of Gibbsimmess of equlibrium states for ç with generalized 
indifférent periodic points. 
Theorem 5 (Characterization of non-Gibbsianness). Suppose that a potential ç 
with Ptop(T, ) <  admits a generalized indifferent periodic point x0. Then there 
is no Borel probabilit meas'ure that is Gibbs for . 

§5. EQUILIBRIUM STATES FOR POTENTIALS OF 'VEAK BOUNDED VARIATION 
Let (T. X, Q) be a transitive FIlS Markov svstem and let ]lIT(X) be the set of 
all T-invariant probability measures on (X,,T). For m E MT(X), I,n denot.es the 
conditional information of Q with respect to T-¢ -. We denote 
JIIT(X,c) := {m E MT(X) I I + 4)  L(m),either h,(T) < oc or 
.,,.O dru > -ec is satisfied}. 
Theorem 6. Let (T X, Q) be a transitive FRS ]llarkov systern ad let dp  W(T). 
Suppose that there ezists a union of full cylinders B(C X) with respect to which 
(T, X, Q) satisfies local ezponential instability and c satisfies local bounded distof 
tion. Let u be the exp[Ptop(T,¢5) - dp]-conforrnal measure supported on X*. As- 
sume further that F := Ç),>0 D, consists of periodic points. If fx* Rdu < oc 
and H,(Q*) < oc, then there ezists a T-invariant ergodic probability rneasu tt 
equivalent to u that satisfies the following variationai principle: 
dp) = bu(T) + .. dp dp = sup{ h,(T) + \. dp dru I m  ]lIT(X, dp) is ergodi«}. 
Ptop (T, 
If ET(X, cP) := {m  MT(X, cp) [ h(T) + fxcpdm = Ptop(T,¢)} contains at 
least two elements, then it iInplies ph.vsically coexistence of different phases, which 
is so-called "phase transition". Phase transition may be related to failure of Ihe 
Gibbs property of equilibrium states (see Theorem 5). 
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Corollary 2 (Phase transition). i'e assume all conditions in Theorera 6. ff F 
consists of 9eneralized indifferent periodie points with respect to ch, then tbe set 
of equilibrium states for ch is the convex hnll of p and tbe set of iTvariant Borel 
probabilil9 meas,u«'es supported ou F. 
In order to prove Theorem 6. we need a sequence of lemmas. Let 3IT*(X*) 
denote the set of ail Borel probability measures on X* invariant under T*. For 
ch* - sR we define 
Mr.(X*,ch* - sR):= {m* G MT.(X*) [ either b,.(T*) <  or 

i.(çb* - sR)dm* > -: is satisfied}. 
Le s0 = Ptop(T,¢). Then P. Walter's lnethod in [14] can apply for T* and for 
* - soRso that there exists the unique equlibrium state ff* equivalent to u and 
the following variatiolm.l principh, is valid: 
0 = r(T*, O* - SoR) = h,,. (T*) + f (* - 
= ,p{,,,*  t.(x*,ç* -.t) I ,«(T') + [ 
(« 
s0R)dm*}. 
« 
Since ,, I "ar. (T «, 0* - 0R) < OE implies the i,ounded distortion property with 
respeçt to u  
Slip Slip slip d(eT.n ) < 
-- t Vj=o T*-(Q* d 
we can show ergodicity and Bowen's Gibbs property for p*. If fx* dp* < OE, then 
the next Schweiger's forlmfla ([13]) gives a T-invariant ergodic probability mesure 
p equivalent to oe that satisfies p(B) = (x-- dP*) - > 0" 

and bv Lemma 5 for f ¢ Ll(p), 

[ Ji\-. -î--(g)- fTi(x) d#* fB, --î=d ()- 
(3)* 
fdlt 
= fx. @* = f, ,d, 
(cf. Lemma 4.2 in [18]). Sin«e fç. Rdp* <  gives the equality 
and Hu. (Q*) < OE, gives b.(T) < ,, we can establish the following characterization 
of the zero so of (GBE). 

Lemma 11. /fp* e Mr- (X*) is ergodic and satisfies ht. (T*) +f\.. (* - «oR)dp 
= O, fx. Rdp* < cx and Ht.(Q* ) < cx, then 

S 0  

,..(T*) + L\-. O*dp* 

f\.. Rdp* 

= h t, (T) + f\ chdp, 

where p is obtained bg fornrula (3). 
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By Lemma 11 we have a T-invariant ergodic probability measure # equivalent 
to  that satisfics Ptp(T, ç) = b,(T) + fx çdl*. 
Lemlna 12 (Lelmna 4.4 hl [18]). g a T-invariant probabilitg measure m sotisfies 
re(B1) = 0, then F := n0 Dn is a full measure set with respect to m. 
Proof of Teorem 6. Bv Lelnma 11 for ail T-ilvalialt ergodic probability lneasures 
m on X with re(B1) > 0 and m  Iç(X. ¢), we can establish 
,(T) + £x- Ca,.- P(T. ) > 
0= 
/I(B1) -- (B1 ) 
On the olher hand, bv Lemma 12, any T-inval-iant ergodic probability ineasure m 
on X with m(B) = 0 satisfies re(F) = 1. In particular, if F := ,,0 D, consists 
of periodic points, then ,(T) + fx dm = J çdm  Pwp(T, ç), which completes 
the proof of Theoreln 6.  

§6. TtlE ('ONSTRUCTION OF O-FINITE ('ONFORMAL MEASURES 
VIA IND[X'ED MAPS 
Let (T, X. Q) be a transitive FRS Markov system and ç 6 çç(T). Suppose that 
(T, X. Q) satisfies local exponential instability and ç satisfies local bounded dis- 
tortion with respect to B1 : j Xj(ff C l). Let .4 = cl(iotXj) for j E J and 
put A := Uj .4.  define the first return function R4 " A   U {} and the 
induced map Ta over { A  R.4() < OE}. Bv the Markov property, there exists a 
partition of the set B A) = {.c ¢ .4  Ra(.r) = k} for each k  1 so/bat T  restricted 
to the interior of each element of the partition is a homeomorphisln onto its ilnage. 
4 denotes the set of all indices corresponding to such elements of the partition 
of ,>1 Bi -A)" Then {v, : i ¢ IA } is a familv, of extensions of local inverses of 
. A()-- çTh(x) _ sR4(x). Bv 
For s Ç N &lld a'e U=I [:A) We defille *}2)(w) = h=0 
Lelmna 4 we can easily see the next fact. 
Lemma la. If each Ai C A satisfles TAs = X the, 
We recall the [ol]owing resu]t in [6]. 
Lm a ([6]). Z I1£,111 < . t   P»(«,é[2 ))  «ti,,, 
,,t{ e - P(4, é ")) e }. 
We suppose that 1 £O1]SSt8 of  sing]e fui] cylhder X and the [o]lowing con- 
ditions are satisfied [or A = 
(05)* 90<< 1,0<<,such that 
and there exists 0 > 0 such that 
(06)* V = (it...iN) E IA and all 0  j < I1,  0 < Lé(i+l...ill) < 
satisfying 
le(%+,...,. (z)) - ¢(v.6+,...., (u))l  L(i+ ...il)d(z,U)  (W.u e A) 
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alld 
I1- 
sup E L4)(iJ+"'ili-I) < 
ilA 
Since the conditions (05-06)* allow us to establish the WBV propertv, of WA a'(»), by 
Theorem 1,  lim, 1 log Zn(¢}}):= Ptop(T4 , CA(»})' where 
n( »'): X X X 
" " I n -!n x:x 
Furhermore, (05-06)* gua.ramee equi-Halder comhmity of {)q :   ln} and 
E% '-(T,  ) < - " i lG,il < . th Gç», .C(A)  C(A). 
Theorem 7 (A construction of -finite conformal measures via induced maps). 
Let (T X Q) be a transitive FRS Markov sgstem and let   (T). Suppose tbat 
th«r« «xts a full cylind«r Xj e Q satis]ying (05)* and (06)* for A := cl(itA3). 
min{O.Ptoç(TA,OA II 
the 
(i) 3so e  with Ptop(4,¢ s°)) : 0 (a g«n«ralized Bowen} «quation); 
(ii) there eists a Borel probability measure oeA on (A.  A) with oeA({X  
A  RA(X) < }) = 1 satisfying E(»oUA = 
(iii) there exi.sts a a-finite measure u on X satisfyin9 £u = [expso]u and 
(iv) in particular, if u is finite, the. Ptp(T. ¢) = so. 

Proof of Theorem 7. B.v Lelnmas 13-14 and Proposition 1, we have the existence 
of s E IR for which Ptop(TA, ¢ )) = 0 and (d  adlnits an eigenvalue 1. Then we 
can apply the main theorem in [5] so that (i)-(iv) are obtained. [] 

The next result gives a criterion of finiteness of u. 

Proposition 5 (A criterion of finiteness of e). Suppose that all assumptions in 
Th«or«m 7 arc satisfi«d. Thon ee(X) : fA exp[s0- ¢]deeA + 1. In particular, if 
infxeA ¢(x) > --oc, then u is finite. 

Proof of Proposition 5. Let I' = {/ [Xt C D N }  First we note the following for- 
mula of u, which was obtained iii [5]: 

Then we see that oe(X) is equal to 

 k 
k=l j_eIA.]jl=k+l v_(A) /=1 
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because of the fact t.hat Xi C TA(V/  I'). By conformality of V A this coincides 
with 

&:l jI.«,lJl:c÷l L (A) 

k 
exp[( Z 0T'- s0)(x)] exp[--0(A°)(z)]&,A(Z) + 1. 

§ï'. THE CONSTRUCTION OF MUTUALLY SINGULAR EQUILIBRIUM STATES 
In this section, we show the existence of mutually singular non-atomic equilib- 
rium states by using induced systems. 

Lenmla 15. Let (T, X. Q) be a transitive FRS Markov system and let ¢ C V(T). 
Suppose that there exists a sequence of full cylinder:s { " M 
Xi}i=I(M _< oe) that sati$fi«s 
(05)* and (06)*, infeA ' ¢(x) > -oc, and 

ffgA rnin{O'Pt°p('TAi 
for each Ai --- cl(bdX). Let Fo := X and for each i > 0 define indu.ctively Fi+I --- 
Ç)n=o T-n(F fq AC+l)(C F). [l'e assume that for each i > O. 
dF 
iCA,+I 
for the Borel prvbability measure 'r,A,+ on Fi  Ai+I obtained in Theorm 7. If 
M 
= F := F con.ists of periodic points, then ther exists a T-ivariant eryodic 
prvbability measure p equivalent to an exp[Ptp(T, ¢) - ¢]-conformal measure  that 
satisfies the following variational principle: 
Ptp(T. ¢) = h(T) + ,. ¢dp = sup{h(T) + f,. d,n , m  Mr(X, ¢) is eryodic}. 
The equilibrium state p for ¢ is not necessarily unique. 

Theorem 8 (Phase transition and singular equilibrium states). IVe assume all 
conditions -in Lemma 15. If F := I___/1 Fi consists of generalized indifferent periodic 
points with respect fo ¢, then there exists a sequence of ergodic equilibrium states 
{#i IM that are mutually singular and the set of equilibrium states for ¢is the 
J 
convex hull of {#}îl and the set of invariant Bord probability measures support«d 
on . 

Lemma 16 (Kac's formula). If f A lAd(oe[A ) < oe and PA is a TA-invariant ergodic 
probability measure equivalent fo oelA, then the next formula gives a T-invariant 
ergodic probability measure # equivalent to ,  

RA (z)-- 1 
#(E)/#(A) =/A Z 
i=0 

lE o Ti(x)d#A(X)(VE e 

Lernma 17 (Finite entropy condition). Suppose that for si := Ptop(TIr,_, 
r._,A,d%_,A, > --oe. Then H,r.«,_ ' (QrA,+,) < 
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Proof of Lemma 17. Let A = Fi ç) Ai+l and s = si. Since we have V3_ E I4. 
eA(Xj)_ =/a exp[¢})(vj(x))]deA(X)" - -- > exp[--s]j]_ evA)inf exp[¢)(x)], 
the bounded distortion for a(°) allows us to see that 
EIA z - 
14 OE IA 
whcrc C is thc bomdcd distortion constant exp[D(1 -)-l(dimnX)°]. These 
incqualitics allow ont to cstab}ish 
H,,A ((A)  [--¢A )dt'.4 + log C + s  H 4duc < OE. 
IA Z 

ProofofLcmma 15. Since n__ Var,(T.4 
,v'A <  is satisfied, it follows from 
Theorem 7 that A, on At satisfying £(s)FAl = F.41 for S = Ptop(T,) and 
A 1 
an exp[Ptop(T,) -]-conformal measure on X. Fnrthermore. bv Proposition 5 we 
see that 
U(Al) -- PAl" The bounded distortion allows Olm to obtain an ergodic TAI- 
invariant probability mesure PAl  PAl with a density dlt 41/doeAl awav froln zero 
(s) 
and infinitv. Furthermore, by [14] there exists an equilibrium state PAl for 
with respect to T4 that is ergodic. In particular, since H,A (Q_41) <  we have 
for 81 : Ptop(T, ), 
top(TAl, CÀï )) 0 l'"Al (1) + ¢81 )d'', ) hA, («1) + 
1 1 
for all a,- invariant probabilitv, measures mA1 e MT, (A, ¢(s))A, (cri [14]). These 
inequalities and Lcmma 16 allow us to bave a T-invariant ergodic probability mea- 
sure g  oe that satisfies lq(A) > 0. pi(F1) = 0 and 
(**) 0 = t,(A,)-(. (T) + .. (¢- 
81)dl) 
for ail T-invariant ergodic probability measures m  3IT(X. ¢) satisfying re(A1) 
0. (**) is equivalent to the inequalities  .s = h m (T)+ f.x- dp  h(T)+ f x- dm. 
On the other hand. any m e MT(X,¢) satisfying t(.4) = 0 is supported on F. 
In fact, since X = (Ui=0 T-A,) u (0(T-A)«). re(A,) = 0 and T-invariance 
of m give m(F) = 1. Thus the set of ail T-invariant probability measures m 
supported on F coincides with the set of ail T-invariant probability measures m 
with m(A) = 0. Since (Tr,,r. Qrl := QF) is a subsystem of (T,X. Q). we can 
apply the above arguments for the induced svstem (%nr,, A2  F. Q%nr, )- That 
 eX rn--1 
is, for s = Ptp(TIr,,¢ ) := lim  log :lfl=  ,=.æer, "Pth=O CT(x)] 
and for the associated potential WA:r, (x) = h=0 - . 
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our assmnptions allow us to establish the i»ounded distortion for 
'.4cr so that 
there exists a 4rl-invariant ergodic probability measure PAr that satisfies 
> h« («r) + 
CA 
2Fl 
for ail 4=r,- invariant ergodic probabi]ity ineasmes m.4=F  ITn=r (A2  FI, 
ç(s=) ). Let p be the T-invariant ergodic prolmbilitv measure supported 
AF 
arising ff'oto It.4p via Kac's formula. Then sz = ht,=(T) + f x dp2  h(T) + 
fx çdm for ail T-invariant ergodic InObability measures m  Mv(A. ¢) supported 
on F that satisfy m(F 5 4z) > 0. Inductively we have a decreasing sequence 
{s}, where s = Pp(Tr_,ç) and a sequ«nce of T-invariant ergodic proba- 
bilitv measures {Yi al 
. }i= such that pi is supportcd on 1" _l.tt,(I'i_  A,) > 0 and 
probability measures m G -lr(X,ç) supported on l'i-1 with m(I'i_  Ai) > O. 
Since yi(Fi-1) I and pi(f/) 0. 
= = {11'}i=1 arc nmtually singular. Filmlly. for 
every T-invarialt measure supported on F that consists of periodic points we bave 
s 2 ,,,(T) + ./v çd,,,. Since {s}e is decreaing, we complete the proof. 
8. APPliCATiONS 
In this section, we show some examples of transitive FRS Markov svstems to 
which our theorems 1-8 can apply. 
Exanple 1 (Brun's map [lai, []8], [0]). ter X = {(.,',,.r) e  "0 < 
1}. and let 
for i = 0.1, 2 where we put  = 1 and z = 0. T is defined bv 
T(Xl,X2) = ( z, z= ) Oll 
1 
T(,FI,3.2) : (x2 1 1) on 
Then Q = {X,}_ 0 is a Bernoulli partition and (0.0) is an indifferent fixed l)oint 
(i.e., det DT(O. 0) = 1). Since T is a continuous piecewise C 2 nmp and a(n) = 
n -1, all conditions (01)-(04) are satisfied and dynamical instal)ility is polylmmial. 
 e that ç = -logdet DT is i)iecewise Lii)schitz continuous so that ç is a 
potential of XVBV. Furthermore, since each periodic point is contained in a single 
cylinder the property (1) is satisfied. Define B = X1 U X2. Then T* satisfies the 
uniformly expanding property and a direct calculation allows us to establish (06) 
for ç = -log[det DT[ (see [18] for more details). Hençe we can apply Theorems 
1-8. In particular, we can see that Ptp(T, ç) = Ptp(T*, ç*) = O. 
Example 2 (InhomogelmOUS Diophantine approximations [13], [15], [16], [17], [19], 
[20], [21]). For the transfornmtion defined in the introduction (Example A), we can 
directly verify all conditions (01)-(04). In fact, we can introduce an index set 
I={(î)  a.bZ,o>b>O, ora<b<0} 
and a partition {X()" ()I}, where X()= {(a',g)  X" a = []- 
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we can veri-IÇ(b) _< log(1 +a( -2)) and a(n)= O(n -1) (see [15], [20]) (cf. [16], 
[19], [21]). Hen«e b = -logldet DTI is a potential of WBV. Since each periodic 
point is contained in a single cylinder, the property (1) in Lemma 6 is satisfied. 
Let D, be the union of cylinders of rank n containing indifferent periodic points 
and let B,, = D,_D,. Thei1 the junlp transformation T*  Ui= Bi  X defined 
by T*(x, ) = Ti(x, ) for (x,9)  B, satisfies exponential decay of dialneter of 
«ylinders (see [19]), and we can veriÇv the validity of (06) for ç = -log[ det DT[. 
Indeed, for  G I* with 1] = , L¢()  3/ 2 and so 
Il- oe 3 
EI* j=O n=l 
Hence we hae smmnability of Var,(T*, *). which allows us to apply Theorems 
1-8. 

Example 3 (A complex continued fraction [51, [12], [15], [22]). For the transfor- 
mation T, dcfined in the introduction (Example B), we define .k'«+ = {z G X  
[1/z] = ,a +'m} for each ,, + m G I := {ma + n" (m.) G Z 2 - (0,0)}. 
Then we bave a countable partition Q = {X},e of X that is a topologically mix- 
ing Markov partition and satisfies (01)-(03). The inverse brmehes to T take the 
forin vj(z) = 1/(j + z), where j G I and the vj satisfy (04). Therefore the inverse 
branches of the nth iterate of the transforllmtion T" take the forin 
,t 1 
Ujl ..... in(Z)- p" + zp.-- and Ivj, ..... j.(z)l = 
q + zq-i ]q + zq-]  
where p = jp_ + Pn- and q = jq_ + qn-, n  1. and p_ = a, P0 = 
0 = q_ and q0 = a. If the string j .... ,j_ corresponds to a cvlinder that 
contains one of the indifferent points, but the longer string j ..... j corresponds 
to a sub-cylinder disjoint from the indifferent periodic points, then ujl,...,j n is an 
imerse branch of the jmnp transformation T* which is uniformly expanding. For 
¢(z) = -log [T'(z)], WBV and (06) are satisfied. Further details tan be found in 
[22] in which multifractal formalism was established by applying our Theorems 1-8. 

§9. APPENDIX- PROOFS OF RESULTS IN §§2 AND 3 
For the proof of Theorem 1, we first verify the following facts. 
Lenmla 18. (18-1) VUk E bl, Zn(Uk, c) > 0 for all n > S. 
(18-2) VU  ,Vn,m > S.+(U,)  (U,)(U,). (S.ubadditivity) 
(18-3) VU.Ut  U. 
z+»,+» (u, )  z(u,é)(c»,c») 
s»--I st --1 
Proof of Lemma 18. (18-1) follows kom Lemma 1 and emark (A) (18-2) %llows 
flore the definition of (E) directly. Since +s,+s(G,) is bounded flore 
below bv 
 inf exp[ 
zXt 
:l[=s.int(TX,, )=U .itXi 1 CUl 
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n-1 
x Z inf exp[ Z çflTh(x)] 
x¢Xj_ 
j:lj l=n,int( T X j )=Uk Dint X  1 h=(J 

X Z xXt 
t_:ltl=s ,int(TXt» )=U ,intXt I CU 
and the transitivity condition allows one to establish 

si--1 
inf exp[ Z 
h=0 

and 

(18-3) follows from the WBV property of ¢. 

Proof of Theorem 1. By (18-2) in Lelmna 18 a.nd the  BV property of ¢, both 
lim ± log(U,¢) and lim - logZ(U,¢) exist for each U  b/. Since 
-- 1 
Zn(U, ¢) _< Zn(U, ¢) _< Zn(U, ¢), by the WBV property of¢, lin,.._,o  log Z,(U, ¢) 
also exists and all the limits coincide. By (18-3) in Lemlna 18, it is obvious that the 
limit does hot depend on U. Noting min<-_<N Zn(¢,Uj) <_ Z,(¢) <_ ;=1Z,(¢,Uj) 
allows one to complete the I)roof. [] 

In order to prove Theorem 3 we first show the next result. 

Lemma 19. (19-1) VV e "12 and Vx  I; £,l-(x) _< _,u,_v Z(U, ¢). 
(19-2) Vx  Uk  H, L:;1u(x) > Z,(Uk,¢). 
(19-3) Z,(¢) >_ 
X()(1) Ç Q satisfles X()(1) ç U ad T(int X(1)) = int X. 

where 

Proof of Lemma 19. We first note tha.t 1T = Uxjcv Xj because of the Markov 

property of Q. Then for x  l 
assertion in (19-1): 

the following inequalities allow us to have the 

jI:X)CV Utld:xUt (ji2...in):TXin=Ut h=0 
Utld:UIDV (ji2...in):TXn:UIDVDX  h:0 

Ut L¢:Ut D V (ji...in ) :T X, =Ut DX 
<_  zo(v,ç). 
Ut lA:Ut Dt" 

n--1 
exp[ Z 
h-----0 
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For a" E Uk, we have the following iuequalit, ies, xvhich give (19-2): 

;lvk(*) = 

jI:X CUk (ji2...in):xTX n 

-> 

exp[- CTh(vji»..,.x)] 
h=O 
h=O 

exp[- ¢h(vji»..i,x)] >_ Z,(Uk, ¢). 
h=O 

By the WBV property of ¢, Z(¢) is |)ollnded fl'oln below bv 

Then (19-3) ff»llows frolll the Markov property and the strong transitivity. [] 

Prvof of Theorem 3. By (19-1,2) iii Lennna 19, we have for x E V C U, 

1 
lira sup - log £lv(x) < 
n --- oc 'Il 

lin, -1 logN( III&X n(Ul,¢)) = Ptop(T, çb) 
n--,oe ?l I</<N 

 log£lu(X) > Ptop(T. çb). Out aSSUlnption Uk ff 12 in,plies U = 
and lira inf,_ g _ 
1". Hence we bave the first assertion. The test of the assertions follow fronl (19-3) 
in Lenmm 19 ilnmediatel b.  

Iii order to prove Proposition 1. we need two sublemmas. 

Sublemma A (Lenuna 3.1 iii [17]). [>0 D, ad X* are positively T-invariat. 
Furtherv,o're, Um=l T*-m(Ç],>o D,) contains ro periodic points. 

Pwof. The result follows from the equality (4)  R(Tx) = R(a') - 1 (R(x) > 2). 

Sublemma B. Deflne 

T'(X, T):= {x X I 3(i...i,0  I" such thatvh...i,x 
7),(X*,T *) := {.c e X I 9(q ...i_,) e I *' such that v,,...iz = x}. 

Pro@ Since x  7),(X,T)ç X* visits B1 infinitely oflen, we Call find a point 
y G 7)(«\*,T *) for some I <_ n such that TJy = :r for solne j < R(y) and 



COUNTABLE TO ONE MAF/KOV SYSTEMS 2969 

l-1 
,=0 R(T*m(Y)) = n. By the property (4) the converse is also true. Since 
I-1 I-1 R(T*m y)-I 
m=0 m=0 h=0 
I--1 
E=o (T*) - 
=  (¢- 
h=O 

we have the rest of the assertion. 

rz=l mT)n(X,T)OX * 

Proof of Proposition 1. By Sublemma A we first note that 7),(X, T) = {7)n(X, T)ç 
fqn>0 D } t2 {P(X, T) C X* }. Then we see that (,¢(exp(-s)) is equal to 
çlo>o»,o(exP(--8)) exp[ exp[-ns] x  exp[n-1 CTh(x)]]  
n=l x(X,T)X* h=O 
Define for  > 1 > 0 E* 
_ ,, := {v e ç,(X*,T*) = 
such that  R(T*'())= , and z = 
Then ç,(X*, T*) = U>t E*.t and ç,(X. T)X* = U<, E,,t. Bv. Sublemlna B we 
see t hat for z  X*, 
.--1 
h=0 

n-1 

We complete the proof. 

= 

ProofofLemma 7. (i) Bv Lemmas 13-14. we have continuity of the flmction 
Ptop(T*,¢* - sR) on int{s e IR I Ptop(T*,¢*) E IR}. Then the existence of a zero 
s0 > 0 of (GBE) follows from the standard argmnent. Since Ptop(T*. çb* - 80/) = 0, 
by Corollary 1 we have so <_ Ptop(T, ¢). If Ptop(T, ¢) < 0, then we bave a contra- 
diction. For (ii), replacing ¢ by ¢- Ptop(T*, ¢*) allows us to reduce t.o the case 
(i). (iii) Since the case 0 _< Ptop(T*, ¢*) < oc is covered by Lemma 7(i), we 
suppose either Ptop(T*, ¢*) = oc or Ptop(T*, ¢*) < 0. If Ptop(T*, ¢*) = o% then 
sup{s e IR" Ptop(T*, ¢* - sR) = oc} = 0. Hence Vs > 0, Ptop(T*, ¢* - sR) < oc. 
Since the function s --, Ptop(T*, çb* - s/) is decreasing and continuons on int{s  
IR ] Ptop(T*,¢* - sR)  IR}, we have lim-m Ptop(T*,¢* - sR) = oc so that for 
suflïcientlly small s > 0. Ptop(T*, çb* - 8) > 0. Oll the other hand, it follows froln 
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Corollary 1 that Ptop(T*, ¢* - sR) < 0 for s > Ptop(T, ¢). Hence we have a zero 
s0 _> 0 of (GBE). le Ptop(T*,¢*) < 0. then sup{.s  " Ptop(T*, ¢* - ) = } = 
Ptop(T*, ¢*). The sanie argmnent as those for the previous case allows us to have 
a zero .% k Ptop(T*, ¢*) of (GBE). 
d(vD = exp[* -- s0R]vi. If 
Proof of Theorem 4. Bv Lemma 8 we have for i G I*, a - 
-  " d ,Z, exp[h=0 Thvi=...i.Z -- son]. Since the property 
(4)  R(Tz) = R(m) - 1 (R(m) k 2) implies a'-.  e I*, the equality 
du - du (ii3---iz)d(m'*=**")du 
now ,, to  tht v& c D,,  ,., = xp[O(,)- 0](V* e X*). O 
the other hand. we know that the above equality holds for Xi C B since i  
I*. Finally. we establish VX, e Q. ].x,(.r) = exp[s0- O(z)](Vz e X*). It 
follows from Lemma 2 and Theorem 2.1 in [18] that s0 = top(T, ). The sertion 
u(Ue 0X)(= u(U,e , &,(X))) = 0 follows from u(i,,lX) = 1. which is obtained 
bv Lemma 8.  complete the proof. 
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STRONGLY INDEFINITE FUNCTIONALS 
AND MULTIPLE SOLUTIONS OF ELLIPTIC SYSTEMS 

D. G. DE FIGUEIREDO AND Y. H. DING 

ABSTRACT. We study existence and multiplicitv of solutions of the elliptic 
system 
-Au---- Hu(x,u,v) in Q, 
--A-v ---- -H,(x,u,v) in Q, u(x) = v(x) : 0 on 
where Q C N, N  3, is a smooth bounded domain and H 
We assume that the nonlinear term 
H(x, , )  I1  + v[ q + B(x, u, v) with lira R(x, u, v) _ O, 
whcre p Ç (1, 2*), 2" :: 2N/(N - 2), and q Ç (1, oe). So some supercritica} 
systems are included. Nontrivia} solutions are obtain. Yhen H(x,u,v) 
is even in (u,v), we show that the system possesses a sequence of solutions 
associated with a sequence of positive energics (resp. negative energies) going 
toward infinity (resp. zero) if p > 2 (resp. p < 2). Ail results are proved using 
variational methods. Some new critical point theorems for strongly indefinite 
functionals are proved. 

1. INTRODUCTION AND MAIN RESULTS 

Cousider the following elliptic system: 
-Au : H(x, u,v) in . 
(E) -Ai, = -H.(x, , v) lu . 
(x) = v(x) = 0 on 

where  C ]K N, N _> 3, is a smooth bounded domain and H   x ]K 2 -- ]K is a C'- 
function. Here H denotes the partial derivative of H with respect to the variable 
u. Writing z :- (u, v), we suppose H(x, 0) --- 0 and H: (x, 0) -= 0. Then z : 0 is a 
trivial solution of the system. Iu this paper we discuss the existence of uontrivial 
solutions. Roughly speaking, we are ,nainly interested in the class of Hamiltonians 
H such that 
H(x.u,v)[u[P+[v[q+R(x,u,v) with lira R(x,u,v)=0. 
I Il » ÷ Il  
where 1 < p < 2* := 2N/(N - 2) and q > 1. The most intercsting results obtained 
here refer to the case when q >_ 2*, v«hich correspond to critical and supercritical 
problems. The case when q < 2* has been studied by Costa and Magalhfies [5], 
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[61 and Benci and Rabinowitz [31. See also Bartsch and De Figueiredo [21, De 
Figueiredo and Magalhàes [7], De Figueiredo and Fehner [8] and Hulshof and van 
der Vorst [11], where similar systems also leading to strongly indefinite functionals 
have been studied. However, only subcritical systems have been considered in those 
papers. 
Letting 2. = 2*/(2* - 1) = 2N/(N + 2), we assume that H(x,z) satisfies the 
following condition: 
(H0) therearep (1.2"), q (1 ) andr  (1, l+q/2.)suchthat, for all 
(., z), 
IH(x. u,v)l  0(1 +11 p- +[vl -1) 

a.nd 

IH,,(x,., v)l < r0(1 + lui p-1 + Ivlq-1). 
In ail hypotheses on 11(z, z) tho 7i's denote positive constants independent of (x, z). 
We note that if q < 2", thon 2. < q/(q - 1), i.e., q- 1 < q/2.. Hence, it is possible 
that q < -r < 1 + q/2.. However, if q > 2", then 7- < q. Furthermore, we relnark 
that 7- can be very large, if q is sufficiently large. 
In addition, we need distinct conditions on H corresponding to the ces when 
p>2, p<2orp=2. 
First. consider the ce when p > 2. In this ce, we assume the following three 
conditions: 
(H1) there are  > 2, u > 1 and Rt  0 such that 
H(x,z)u + H.(x.z)v 2 H(x,z) whenever Izl  R1, 
with the provision that v = g if q > 2: 
(H2) there are 2.(p- 1)    p and 2.( - 1) <  such that 
H(,z) OE l (ll  + Il ) -- 2 O all (,z). 
and  = q if q > 2*; 
(H3) H(,0, v) OE 0 and H(z,u,0)= o(11) as u  0 unifor,nly in z. 
ç prove the following results. 
Theorem 1.1. Let (Ho) be satisfied witb p > 2. If (Ht) - (H3) hold. then (E) bas 
at least one nontrivial solution. 

In order to provide some more transparent hypotheses under which the above 
result holds, we next present some conditions on H that are sufficient for (Ho), 
(Ht) and (H2) to hold: 
(H) there are p  (1, 2*) and q  (2, 3c) such that, for ail (x. z), 

and 

IH,,(x,u,v)l <_ "),0(1 + lui p-1 + Ivl §-1) 

IH(oe, u,v)l <_ o(1 ÷ Izl p-1 4- 

(H[) there are/ > 2 and RI OE 0 such that 
H=(x, z)u + Hv(x, z)v > ¢tH(x, z) 
(H) for p and q as above, 
H(x. z) >__ ")'1 (11 p + Ivl q) -- "r2 

whenever Izl >_ 

for ail (x, z). 
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Theorem 1.1'. Let (HIc) be satisfied voith p > 2. If (H[),(H;), and (Ha) hold, 
then (E) bas at least one nontrivial solution. 
Theorem 1.2. Let (Ho) be satisfied voith p > 2. If H(x,z) is even in z and 
satisfies (H1) and (H2), then (E) bas a sequence (z,) of solutions with eergies 
I(z) := t (}(IVul  -]Vvl u) - H(z,z)), going fo oe as ,,  . 
In order to describe the other results, let «(-A) denote the set of ail eigenvalues 
of (-A,H()): 1 ( 2  3  "" ". 
We now consider the case whcn p < 2.  makc the following assumptions: 
(Ha) there are it G (1, 2), u OE 2 and 3 OE 0 (3 = 0. if q > 2*) such that 
H(x,u,v) 2 H(x,u,v)u+ H(x.u,v)v-3 for all (x,z); 
(Hs) there are a  (1, 2)and 5  (0, 1/2)su«h that H(x.u,v)  ?41u[ a -SAv 2 
for all (x, z)" 
(u) if q  2-, th« U,.(z, ),  ,[ -(v + ) fo 1 (z, ). 
With these assmnptions we have the following three results, for the cae when p < 2. 
Theorem 1.3. Suppose that (Ho) holds with p < 2 and q  2. If H(x,z) also 
satisfies (Ha) - (H), tben (E) has at least one nontrivial solution. 
Theorem 1.4. Suppose that H(x, z) is even in z and (Ho) holds with p < 2 and 
q  2. IfH(x,z) also satisfies (H4)-(H), then (E) has a sequence (z) ofsolutions 
with negative enewies I(zn) going fo 0 as n  oe. 
Theorem 1.5. Let (Ho). with p,q  (1, 2), and (Hs) be satisfied. Then (E) bas 
at least one nontrivial solution. If. in addition. H(x, z) is even in z, then (E) bas 
a sequence (zn) of solutions with negative energies I(zn) going to 0 as n  oe. 
Finally, we consider the ce when p = 2, which presents some sort of resonance. 
Assmne 
(HT) there exist b0 _< 0 < a0 such that Ro(X, z) := H(x, z) - (aou   + boy 2) = 
o(z] 2)  z  0 uniformly in x; 
(Hs) there exist «  (1, 2), aoe  la0, Oe)«(-A), such that R(x,z) := 
u(, )-  t« 10(, )1  (l+ll-+vV -) o n(, ) 
 lv-(1 + 
The position of the numbers a0, aoe, b0 with respect to the spectrmn «(-A) plays 
a very essential role in the next result. For that marrer, let i,j, k be nomegative 
integerssuch that Ai =min{A  «(-A)  A > a0}, A =max{A «(-A)  A < 
{j if a = a0, 
[= j+k-i+l if a > a0. 
Now we can state out last result. 
Theorem 1.6. Let (Ho) be satisfied with p = 2 and r < 1 + q/2. Assume tbat 
u(z,) i v« i  ad atifi () d (gs). Th (E) ha at tat o« pai of 
nontrivial solutions if  = 1, and infinitely many solutions if   2. 
The ces covered in Theorem 1.6 include some ymptotically linear systems. 
Such systems bave been studied in [5], [6] and Silva [13]. However, their results are 
hot comparable with the ones obtained here. 
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We organize the paper as follows. In order to establish multiplicity of solutions we 
need some new abstract propositions on critical point theoD- for strongly indefinite 
functionals, which will be provided in Section 2. These propositions are based on 
certain Galerkin approximations, and we emphasize that the functionals do not 
satisfv the usual Palais-Smale condition. In Section 3 we study svstems that are 
_-uperlinear in the variable u. and prove Theorems 1.1 and 1.2. In Section 4 we 
consider systems that are sublinear in the variable u. and prove Theorems 1.3. 1.4 
and 1.5. In both Sections 3 and 4. the variable v can have subcritical grooEh as well 
as supercritical grooEh. Finally. in Section 5. we consider a special asymptotically 
linear svstem and prove existence of multiple solutions. 

"2. CRITICAL POINTS FOR STRONGLY INDEFINITE FUNCTIONALS 

Let E be a Banach space with norm [[-[[. Suppose that E has a direct sure 
decomposition E = E  @ E 2 with both E  and E 2 being infinite dimensional. Let 
P denote the projection from E onto E . Assume (e) (resp. (e)) is a basis for 
E  (resp. E2). Set 
X := span{e, ..... .en} + E 2. ,\' := 
and let (X) ± denote the complement of X m in E. For a flmctioIml I  Ci(E, 1) 
we set In := Il \,, the restriction of I on \-n- Recall that a sequence (z j) C K is 
saidtobea (PS) sequenceifzj  Xn n --. _-'c, I(z)  c and I' (--) Oas 
j  :. If an)" (PS) e sequence bas a convergent subsequence, then we sav that I 
satisfies he (PS) condition. 
Denote the upper and lower level sets. respectively, by la = {z  E " I(z) _> 
a}. I  = {:  E" I(z) <_ b} and I =/ ç I  (denote similarly (I)«, (I)  and 
/)). Wealsoset K = {z < E- I'(z) =0}. /Q = Kç/. K  =KçI and 
/C =/Q c/C . 
Proposition 2.1. Let E be as above and let I  C(E, IR) be even u, tth I(0) = O. 
In addition, suppose that. for each m Ç N. the conditions below hold: 
Ii) there is Rn > 0 such that I(z) < 0 for all - 
,12) there are r, > 0 and a, --  such that I(z) > a, for all z Ç (X"-I) ± 
with Ilzl[ = rn" 
113) I is bounded from above on bounded sers of X': 
I_ 0 if c >_ O. any (PS) sequen«e (:) has a subsequence along whi«h zn -- z 
Then the functional I has a sequence ( ck ) of critical values, with the property that 
Ck  -. 

Remark 2.1. This proposition is more or legs known if the condition (I_) is replaced 
bv the (PSt* condition (cf. [1], [9]), or bv the usual Palais-Smale condition, that 
is. any sequen«e (:k) C E such that [I(zk)[ <_ c and I'(zk) -- 0 bas a convergent 
subsequ«nce (cf. [31). 
Proposition 2.2. Let E be as above and let I  C(E.I) be even. A.sume that 
I(O) = 0 and that, for each m  N. the two conditions below hold: 
115) there are r > 0 and a. > 0 such that I(z) >_ ara for all :  X m with 
ri6) there is b >0 u'*th bm -- O uch that I{:) <_bru for all :  (X-) ±. 
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Moreover, suppose that either I satisfies the (PS) condition for ail c > O. or that 
the condition below holds: 
(17) inf I(/Ç) = 0. and. for allc >_ O. any (PS); sequ«nce (z,) bas a subsequence 
alon9 which zn  z  Çc witb z = 0 only if c = O. 
Then I bas a sequence (ck) of positive critical values satisfying Ck  O. 
Proof. Let E be the family of symmetric, closed subsets of E{0}, and let   E  
N U {0, } denote the Krasnoselski genus nlap. Set 
% := sup 

where 

rU := {A  r- A c X a,,d (A) >. + ,,,}. 
Fix m  N. The Borsuk-Ulam flmorem implies that A  (xm-) ± 7  for eaçh 
A  Z TM. It follows from (Ia) that 
inf I(z) < sup I(z) < b,. 
zA z( k'm- 1 ) / 

On the other hand, since 
and so, by (I5), we obtain 

Therefore, 
(2.1) 

çX, ) = n+m. one bas S,, " := OBr,f-)X,rî E E n , 
inf I(z) >_ ara. 

ara <_ c n _ 

A standard deformation argument, using a positive pseudo-gradient flow. yields the 
?72 OO ?72 
existence of a sequence (z)=, with z  X satisfying 
1 1 
[I(z) - cl  and , m 
? 71 
 «an sun,e that I(z)  cm as n  . So, (z.,) is a (PS) sequen«e with 
(2.2) a  c  b. 
Now, if we assume that I satisfies the (PS)gcondition for c > 0, then the conclusion 
m 
follows. Next, suppose instead that (I7) holds. Then, along a subsequence, z 
z as n   with I'(z) = 0 and 0 < I(zm)  Cm. Finally, by (2.2), 
I()Sb o, 
and the proof is complete. 
Proposition 2.3. Let E be as above and let I  C (E,N) be even wth I(O) = O. 
Suppose, in addition, that the three conditions below hold: 
(Is) the «e e N .,  > 0 h tht I() 
(I9) there is b > 0 such that sup I(E 2)  b: 
(Ira) any (PS), c > O. sequence (z) bas a subsequence along which z 
Çc and Pz  Pz. 
Then I bas at least one pair of nontrivial critical points if  = 1. and infinitely 
many ctical points if f > 1, with positive critical values. 



2978 D. G. DE FIGUEIREDO AND Y. H. DING 

Pro@ Let E, % E T and c be as in the proof of Proposition 2.2. As before, by 
(I8) and (I9), we obtain 
" < b for all n  N and tri = 1,.-- ,. 
and we find sequences z G X such that, going to subsequences if necessary, 
I(z)  c and I,,(z n )  0 as n  , with 
b  Cl  C2  "'"  Ct 2 a. 
Using (I0), we can assume flrthermore that n'  z G « for m = 1,--- , g, as 
n  . If g = 1 the proof is complete. 
Consider (> 1. Let F = {z G " I(z) > 0}. aregoingtoprove that F 
is an infinite set. Arguing by contradiction, we suppose that F is finite. Çhoose 
0 < g < a  b < u satisfying 
tt < inf I(F)  sup I(F) < u. 
Let k G N be so large that 0  A := QF, where Q  E  X  denotes the 
projection. Then A is also finite, and (A) = 1. By the continuity of % for all 
5 > 0 small, (N(A)) = (A), where N(A) = {z  X   dist(z,A)  5}. Set 
c = (.4)(x) . Sn« (A) c c ,,d Q "C  V(A). t onow rom th 
properties of  that (C6) = (N(A)).  remark that Q = p1 + (Qk _ p) and 
that the range of Q - p1 is k-dimensional. So bv virtue of (I10), we conclude that, 
for all c  0, any (PS) sequence (zn) has a subsequence along which zn  z G « 
and Qkzn  Qkz. Hence there are n0 G N and a > 0 such that for all n  n0, 
[]I,'(w)l ]  a for all w G (I)  C, 
where C = C6  X. Bv a standard deformation argument, we can then construct 
a sequence of odd homeomorphisms q  X  X such that 
,, ((&) k C?) C (Z,) 
(cf. [12]). For n0 sufficientlv large, we can suppose that 
p<% % _'"_c n<u forallnn0. 
Let G G E be such that inf I(G) > ( + c)/2. One then h 

and 

(,»,(c\c')) = (c\c') >_ (c) -(c') 
>  + t. - ")'(Ce) > r, + t. - . 
Thus rl,(G\C')  Z -1 and v <_ inf I(rl,(G\C') ) <_ %t-,. One finallv, con,es to 
t--1 
r, _< % < r,, which is a contradiction. [] 
From now on we turn to the system (E). We denote by I" It the usual Lt(Q) norm 
for ail t  [1, oc]. For q > 1 let Vq = H(Q) if q <_ 2* and Vq = H(Q) çlLq(Q), the 
Banach space equipped with the norin IIvllvç -- (IVvl ÷ IriS)1/2, if q > 2*. Let Eq 
be the product space H(t2) x I with elelnents denoted by z = (u, v). We denote 
the norm in Eq by Ilzll« = (Iwl ÷ Ilvll) 1/2.  has the direct sure decomposition 
Eq = E @ E +, z = z- + z + 
where 
E-={0}xVq and E +=H(t2) x{0}. 
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For convenience, we will write z + = u and z- = v. t/eca.ll that by (A,),EI we 
denote the sequence of eigenvalues of (-/N,H{II(H)). Let e,, le,12 = 1. be the 
dgenflmction corresponding to A, for each n E N. Clearly, e + := (e,, 0), r E N, is 
a basis for E +, and e = (0, e,), n  N, is a basis for E-. 
Suppose that the assmnption (Ho) holds. Then 

(2.3) 

So the functional 

(2.4) 

(2.5) 
and 

H(x,z) <_ c(1 + lu[ 2. + I¢,1 ) o n («,z). 

il £ 
I(z) :=  ([Vul 2 -IVvl 2) - H(x,z) 

(2.6) ] H,,(x,z)b  ./o H,(x,z)b for ail 
By the Sobolev elnbedding theoreln and using interpolation, we obtain that 
u  u in L t for t  [1, 2*) and v  v in L t for t  [1, q). Noting that 
]H(x,u.v)  %(1 + u -I + ]v -) with 2.(r- 1) < q, (2.5) follows easilv since 
u in L ,v vinL 2.(-1) andç H() C L 2.. Next weseethat (2.6) 
is clely true when ¢  L . In general, for a ¢  t we proceed as follows. Let 
ç e L oe with   ¢ in L q as m  oe. So 
IZ(H(x,z) - H,(x,z))¢I = I£(H,(x,z) - H,(x,z))( + (¢- m))  , 
and using (Ho) we see that this expression is less than the following sure: 
+ , (> - ml + >vlm -', + va-m - 
which by its turn is estimated bv 
since (z) is bounded in Eq and L  is dense in Lq. So (2.6) is proved, and it follows 
that 
I'(z)w + I'(z)w for ail w  Eq 

is well defined in Eq. Moreover, I ¢ Çl(E'q,]l), and the critical points of I are the 
solutions of (E). 
Lemma 2.1. If (Ho) holds, then I' is weakly seqttentially cotinuous, hat is, 
I' ( z, ) -- l'(z) provided z, -- z. 
Pro@ If q < 2* this statenlent is well known. Assume now hat q > 2". Let z, -- z 
in Ev Çlearly, for all w = (ç, )  Ev we bave 
So it renmins to show that 
[ H=(x,z,)o -- [ H=(.r,z)o for all qO  Ha(t ) 
J 
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3. THE CASE p > 2 
Throughout this section let (Ho) be satisfied with p > 2, and assume that 
and (H2) hold. Observe that, by (H2), there exists R > () such that H(z,z) > 0 
whelever Izl > R. This, jointly with (H1), implies 
(3.1) H(x,z) > «(lui" ÷M)-c2 foran 
(see [10]). This, together with (2.3) and (H2), shows that 
(3.2) u <_ q and /3 _< q. 
Moleover, by virtue of (3.1) and (H..,), we lnay assulne, without loss of generality. 
that (since p > 2) 
(3.3) « > 2. 
1 e 2 e + for ail n (5 N. So 
Now we set. E 1 = E-, E 2 = E + and e n = , e n = 
Eq = E1 ( E2. Consider the flmctiolml defined by (2.4), which has the properties 
stated iii Section 2. 

Lemma 3.1. ,4ny (PS) sequence is bounded. 

Pro@ Let z, G X, be such that 
I(z,)  c and I'(z,)  O. 
--   n), we 
Cae 1: q < 2. h thi ca E = (n()) . m" (n), for ' := ( , 

have 

(3.4) 

=(_71 _ 1, )lVul ÷ ( 1 _ __ )lVvl 

÷  (;H"(x'zn)un + 1--Hr(x'zn)vn - H(x'zn)) -cloe 
1 £ IwI 1 1)lwl-c» 
->(- ) +(;- 
If q < 2. then (3.2) shows that pe < 2, and so I1=11 -< e(a ÷ I1=11), which implies 
that (z,) is bounded in Eq. Assulne q = 2. hvokillg (3.2), we get u  2. and so 
I  c(1 + [lz[lq) by (3.4). Since ti(x,z) > 0 for all [z] large, and 
one sees that [z[[ N c(1 + [[zllq). Hence, (z) is bounded. 
Case 2: q > 2. Note that in this case  = p > 2 in (H). So 

(3.5) 

I(z,)- -I'(z,)z = (7_H(a',z)z- H(x,:)) 
# - 1) f H(x, z) - 
_>( 
C, 

which, together with (H2), yields 
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Using (Ho), we get 
(3.7) 
Next we estimat.e the integrals in the right, side of (3.7). Since 2.(1»- 1)    p, 
we have that 0 := /(1 + a - p)  2*. Using tho H61dor inequality, the Sobolev 
embedding theorem and (3.6), we obtain 
Similarly, since z - 1 < ffC., we bave 1 < w := /(1 + [3 - z) < 2*, and hence 
 Il-ll[  I' - 
Therefore, usiug th  est.ima.l«  in (3.7), wc oh/aih 
_ , l+(p-)/ 
Iç,t < «(1 + It  + IIzll+<-')/n)  

(3.s) 
Since 

IVv,] = -l'(z,)(O, v,) - fa H-_(a', z,)z, + fa H,,(x,z,)u,, 
and using (3.5) and the above argmneuts, we obtain 
(3.9) IV,l 
Recall that, in view of out sumptions, (p- 1)/  1/2., (r- 1)/ < 1/2.. and 
fl = q if q > 2*. Hence, it follows from (3.6) and (3.8)-(3.9) that (z) is bounded 
in Eq. 
Lemma 3.2. Let z,  X,be a (PS) sequence. If q  2", then (z) contains 
a convergent subsequence. If q > 2", then te is a z  Eq such tat. along a 
subsequence, z  z and l'(z) = 0 and l(z)  c. 
of. By Lemma 3.1, (z,) is bounded.  can assmne that z,  z in Eq, z  z 
in (L()) 2 for ail 1  s < 2*, and z(x  z(.r) a.e. on . It follows from the 
weak sequential continuity of I' (see Lemma 2.1) that l'(z) = 0. Since I'(z)  0, 
we obtain 
). 
Using (H0) and the H61der inequa.lity, we obtain the estimate 
where  is as in the proof of Lemma 3.1. Hence I1  Iul, -hih implies 
  
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in Eq for ail z E Eq. Moreover, using again (Ho) and the H61der inequality, we 
estimate 
_< c([ç -- Pnçll + [n[ç-ll ç - Pç[ç+ ]ç[ç-l[ t' - Pat'[q) + 0. 
On the other hand, 
I[(z)(O,v - Pv) + [. H,,(.,z)(v- Pv) 
VV)L 
+ 
Lebesgue's theorem and the weak sequential continuity of H:(«,-) (see the proof of 
Lemma 2.1) yieht 
] Vv, - liln sup ] Vv ] = linl inf ( £ H: (,r , z ) zn - £ He ( x , z ) Z )noe noE  0, 
i.e., IVvl 2 lira sut,,, V*'I. This, together with the weak lower semicontinuity 
of norms, implies Vvu + [Vvlu. So v + v in H(). 
Therefore, if q  2*, we obtain that, along a subsequence, z + z in Eq and 
consequently I(z) = c. Next a.ssume that q > 2*. Observe that 

hence, 

Lebesgue's theorem then yields 
I(z) -c= liif  H(x,z,O - £ H(x,z ) >_0, 
that is, I(z) >_ c. 
Lemma 3.3. If(H3) also holds, there are r,p > 0 such that inf I(OBrE +) > p. 
Pro@ By (Ho) and (H3), for anv e > 0. there is c« > 0 such that 
H(.. . 0)< el,[ 2 + c«M 2". 

Hence, 

1 
and the conclusion follows easily. 
Let e E E + with IVel = 1, and set 
Q= {(se, v)  0 _<  _< q, Ilvll <_ r2I. 
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Lemma 3.4. If(H3) also holds, there are rl,r2 > O, wzth rl > r, such that I(z) <_ 0 
for all z Ç OQ. 
Pro@ By (H3), I(z) <_ 0 for all z (5 E-. By (H2), 
I((se, v))< 2 [7v]-c1 (lel÷lvl)÷c2 
The conclusion follows since  > 2.  

We are now in a position to prove Theorem 1.1. 

Proof of Theorem 1.1. Lemmas 3.3 and 3.4 soEv that I has the linking geometry. 
Let Qn := Q 0 Xn, and define 
ch := inf maxI(3,(Qn)), 
where Fn := {? e C(Q,X,) : /IOQ, =id}. Then p_< c, _<  := supI(Q). A 
standard deformation argument shows that thcre is z E X such that II(z)-cl < 
lin and I{I.(Zn)[I _< l/n. So we obtain a (PS): sequence (Zn) with c e [p, c]. 
Lemma 3.2 implies z -- z with I'(z) = 0 and I(z) >_ c. The proof is complete. [] 

We now consider the multiplicity of sohltions using Proposition 2.1. 

Lemma 3.5. I satisfies (Il)- 
Pro@ Using (H2), we obtain 
I(z) - z < lVul -  1]Vv] -c  (lui a + Iv] ) + 
C 2 . 
Since all norms in span{e,.-- , e,,} are equivalent, we obtain 
I(z) <_ -(«31Vu] -2 )lVu]- (-]Vv]-- «l]V]) --C 2, 
for ail z = (u,v) G X TM -- span{el,-.-,e} x 1. So (It) follows easily. 

Lemma 3.6. I satisfies (I2). 
Pro@ Since (X) ± C H(ft) and H(ft) embeds compactly in LP(ft), we have 
that r > 0 and r  0 as m  o, where 
(3.10) rh := 
sup ; 
see Lemma 3.8 in [14]. For z = (u, 0)  (X') ±, it follows fi'om (Ho) that 
 1 iÇtl2 2 _ ci itl p _ C 
I(z) = 1 IVl- H(x,,O) _> - 
1 
-2 
Setting r, = (pcr) 1/(2-») and a, = (p - 2)r/2p - c2, we corne to the desired 
conclusion. [] 

Proof of Theorem 1.2. Since H(x, z) is even in z, I is even. Lemma 3.2 shows 
that I satisfies the assumption (I4) of Proposition 2.1. Lemmas 3.5 and 3.6 show 
that (I) and (I2) hold. Clearly (I3) is also true. Therefore by Proposition 2.1, 
there is a sequence (z) C Eqsatisfying I'(z) = 0 and I(z,) -- o. The proof is 
complete. [] 
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4. THE CASE l» < '2 
Throughout this section we assume that (Ho) is satisfied with p  (1. 2). 
also suppose that (Ha) - (H) hold. 
Let Eq = E 1  E 2 be as in Sectim 3. Consider the flmctional 
Lelllllla 4.1. Auy (PS) scqu.«m'e (Zn) ha.s a subsequence collverging wea£1y to a 
criti«al poi'nt z qf J with .I(z)  c. a,d z = 0 only if z=  (1 i Eq. 
Pv@ The l)r()of is divi(h,d ino wo parts. 
Pat 1. The sequ(,n('c (z.) is bom.led in Eq. By (Ha) it follows that 
Hence IVu,,l  c(1 + IIzllq). If v > 2, we also got IV,l  c(1 + llzllq). If v = 2, 
wc uso (fla)and tire fa«t that Içl OE a,ll i, or,ie to obtain 
Thus, if q N 2*, thon (z.) is bounded in Eq. Assmne next that q > 2*. It follows 
ri'oto (Hi) t hat 
in Eq also in the case when q > 2*. 
Part II. %% can now suppose that z.  z in Eq, On  Z in (L(ll))  for all 
1 N s < 2*, and z(m  z(m a.e. in m Q. It follows that z is a critical point of 
J. As in the proof of Lonnna 3.2. using (H0) and 

Using (Ho), we have 
/-/,, (.r. 

+ fa Ho(x, z.(v - P. 
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Colsequent ly, 
(4.2) ('v, (v-- v,))L, = i Hr(.'r, Zn)( -- v) + o(1 ). 
Thus if q < 2", it follows from (4.2) that {Vt,[  
and so zn  z. This proves lhat .l satisfies he (PS);condition in this case. and 
that J(z) = c. 
Consider next q k 2*. The weak sequential contilmity of H,,(.r,-) (see the proof of 
Lemma 2.1) yields fn H(.r, zn)v  j; H,(.r,z)v. By (H), fn(z):= H,(.r, zn)t' + 
76(1v1+11 )  0. Using the fact that [v]  ]v[ and [t,]  1[, and applying 
Fatou's lemma to the sequence (f), we get 
Using this estilnate in (4.2), we obtain that IV,]  lilnsUp,,oe V'I, whi«h 
implies that. t,  ' in H). In Ol'dm to Colwlud« that .lz)  c, we use the 
est ilnat.e 
.l(z)-.J(z) =  (H.r, zn)- Ha',z)) + o(1), 
and so z  0. 
Remark 4.1. In a similar way, using even silnpler argulnents. OlW checks that. if 
(H0) holds with p, q  (1, 2), .1 satisfies the (PS) condition for all c. 
Remark 4.2. Let ,], = J x- denote the restriction of J on X . As in Lemlna 4.1, 
it is hot difficult to check that. if the sequence (z) Ç Eq, with z G X , satisfies 
J(z)  c and (z)  0 as m  , then it possesses a subsequence converging 
weaklv to a critical point z of J with J(z)  c. and z 
also bave, as in elnark 4.1, that. if (H0) holds with p. q G (1, 2), then anv such 
sequence bas a convergent subsequence. 
Lemma 4.2. There is an R > 0 such that J(z)  0 for all z = (v,O) with lzll 
ool. y (,,), w hv (,,..0)  «(1 + ). 
and the lelllllla follows, SillCe p < 2. 
Lemma 4.3. Fore > 0 small the is p > 0 such that J((ee.v))  p for all 
v  . whe el is the eigenfunction correspodin9 to the first eigem, alue  of 
(-.  (t)). 
Pro@ Bv (H.5), for e > 0 small. H(z, ee,v) k 24eeî- akv2: hence, 
, 1 'V'" -  > ('[e" -  '-)" 
J((ee.v)) = H(.r, ee. v) +  _ 
The conclusion follows. 
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are llOW ready to prove Theoreln 1.3. 

Proof of Theorem 1.3. Recall that X TM  span{el,--- , e,} x l,q, and consider the 
restrictions J, as defilmd iii Rema.rk 4.2. Set Dn = Bn ç E 2 = Bn fq (HH(12) x {0}) 
and Dru = DR fq X m, where R > 0 cornes from Lemma 4.2. Define 

ç,n := inf max J(7(D,n)), 
where Fro := {7  C(D,, S') : 7(z) = z forall z  OD,,}. It is well known 
tiret 7(Dru)  II" ¢ ¢ for all 7  Fro, where II" = {(«o.0)} x 1 with « > 0 
slnall. Invoking Lemma 4.3, we fix an ê > Il so snmll that there is p > 0 satisfying 
inf./(ll')  p. Then we have 
t'  c.  b := max J(Dn). 
The well-known saddle 1,oint lheorem (cf. [12] or [4], [14]) implies that there is 
z, G X m satisfying ]J(zm) - c]  l/m and ]].[(zm)]]  1lin. Now by virtue of 
Remark 4.2, along a sul,sequence, z  z with ,l'(z) = 0 and z ¢ 0, ending the 
proof.  

"Ve llOW turn to the proof of Tlworelns 1.4 alld 1.5. 

Lemma 4.4. If. in addition. ")'3 = 0 in (Ha), then J satzsfies (I5)- 

Proof. It follows from (Hs) that 
J(z) > c I I + ( 
(4.3) 
Since a < 2, the result follows in the case when q  2*. Next consider q > 2*. 
Suppose (I) does hot hold. Then for any r > 0 there is a sequence zj  X TM such 
that IIzll =  al,d J(z)  O. It fonows frein (.a) with z = z, and for r slnall. 
that Içl=  0 and [çvj[=  0. Ail this ilnplies that 
sumption (H0) and t he fact t hat (ai) lies in a finite-dilnensiolml subspace, it follows 
that fH(x, zj)uj  O. Colasequently, by (H4)with 73 = O, f Hv(x, zj)vj  O. 
This, jointly with (Ho), yields 
0. 
Hence, zj  0 in Eq, which is a contradiction. 

Lemma 4.5. J satisfies (16) 

Proof. By (Ho), H(a:.,,,0) <_ c(lu[ + I**l), alld so, for u  (X'-I) ±, Olle hg8 
 (Cll']p- [ 
l[v[)lVu[ = + (Cl  _ 1 
where Vin was defined by (3.10). Let b := 
Then 0 < b  0 and J((u,0))  b for ail (a,0) 



STRONGLY INDEFINITE FUNCTIONALS AND MULTIPLE SOLUTIONS 2987 

Proof of Theorcm 1.4. Since H(x,z) is even in z, d is even. If q < 2", then d 
satisfies the (PS) condition for all c (see the proof of Lemma 4.1). If q > 2*, then, 
using assumption (H4) applied to a critical point z, we obtain 

3(z) = 3(z)- £(z)(**,-v) > 1 1 Iwl + 

0. 
_'2 ,t  - 

This, jointly with Lennna 4.1, shows that (If) is satisfied. It follows from Lennnas 
4.4 and 4.5 that J satisfies (I5) and (I). Therefore, the desired conclusion follows. 

Finally, we prove Theorem 1.5. 

Proof of Theorem 1.5. The proof of the existence of one nontrivial sohltion is sim- 
ilar to that of Theorenl 1.3, using Ptemark 4.2 and LellllllaS 4.2 and 4.3. The 
other conclusion tan be obtained along the lines of the proof of Theorell 1.4, using 
Remark 4.1 and Lennlms 4.4 and 4.5. [] 

5. THE CASE p = 2 

hl this section we alwa,vs assume that (Ho) holds with p = 2 and r < 1 + q/2. 
We also suppose that (H7) and (H8) are satisfied. We will apply Proposition 2 3 
in order to prove Theorem 1.6. Thus, set 

/72 =span{eï, --- ,e} : /I ----spall{el,'--,et-} X Ç, /71 - /q(E '2, 

and 

X £ /71 (t)span{ï,-- + (7,"" (;} 
  ,ek, ,  

2 + forl<< 
1 + for n E 1, and e n= en+i_ 1 
One may arrange the bases as % = %+ _ _ 
2 + for g < n < f+i- 1, and 2 - 
2 - forg-j<n<t e . %=%_t %=e,_ 
g--j, e n = en_t+ j -- -- 
for rt > f + i- 1. Consider the filnctional I given by (2.4). 

Lemma 5.1. I satisfies (Is); that is, there exist r, a > 0 such that I(z) >_ a for all 
z e X t with Ilzll = - 

Pro@ Let z = (u,v)  X e. Since v Ç span{el,.-- ,ej}, we have v Ç L . By (Ho) 
and (H7), for any e > 0, there exists ce > 0 such that 

/0(, ) < ell 2 + c,(M 2" + Iris). 

Thus 

1 
I(z) =  (IVul - a, olul ) - - ([x7vl - b01vl) - J 
Ro(x, 
Z) 
'(_ )  + ( ),-«- (: + 
> _ ao V 2 1 -bo 
-2   

Now the conclusion follows easily. 

Lemma 5.2. I satisfies (I9); that is. sup I(E 2) < oc. 
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Pro@ For z E E 2 we ha'«', using (Hs), that 
 IV'loe- f/(', z) 
[() =  
- '2  
which ilnplies that l(z)  0 for ail z e E 2 with IIz[[«large. 
Lenlma 5.3. Let c > O. Thcn any (PS)c sequewe ts bounded. 
Pv@ Wc decompose H] (t)  
H(t)=U-6 +, t,=,-+u +, 
«hcre - = span{e,---,eh} and U + is the orthogonal complelnent of U- in 
,l(t). 
Let (z) bc a (I'S)g sequence. Using the expression of I,': 
  +2 
plus (Hs) and the Hi;lder inequality, we obtail 
a--1 + 
A+ - 
where r = q/(l+q-ç). By sumption. 1 < r < 2. It then follows Kom the 
Sobolev embedding theorems that 
+ - 
Similarly, we deduce that 
The two previous inequalities ilnply the estinmte 
Using the expression of H given in (Ha), and recalling that I(z) > 0 for large 
we obtain 
(5.z) }   .- -_ 
Next using (5.2), assulnption (Hs) and (5.1), we obtain 
The combination of (5.1) and (5.3) implies 
+ 
Since a < 2 and 2(r - 1) < q. we see that (z,,) is bounded. 
Lemma 5.4. I satisfies (Io). 
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Pro@ Let (zn) be a (PS); sequeuce wiîh c > 0. Using Lelmlm 5.3, an argulient 
similar to that of Lçlnlna 3.2 shows that along a subsequence zn --" z E/Cc, we bave 
,, -- z in Hot(tl). Since E1 C H01([), we have Pzn  Ptz. [] 

Pwof of Theorem 1.6. Since H(x, z) is evell in z, I is even. Bv assmnption. I(0) = 
0. Lemnlas 5.1, 5.2 and 5.4 show that I satisfies (Is) - (It0)- Now Proposition 2.3 
applies, aiM the proof is COlnplete. [] 
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STABILITY OF SMALL AMPLITUDE BOUNDARY LAYERS FOR 
MIXED HYPERBOLIC-PARABOLIC SYSTEMS 

F. ROUSSET 

ABSTRACT. We consider an initial boundary value problem for a symmetriz- 
able mixed hyperbolic-parabolic system of conservation iaws with a smail vis- 
cosity e, u[ + F(u«)x = e( B(u «)uex)x. When the boundary is noncharacteristic 
for both the viscous and the inviscid system, and the boundary condition dissi- 
pative, we show that u « converges to a solution of t he inviscid system before the 
formation of shocks if the amplitude of the boundary iayer is suflïciently smaii. 
This generalizes previous results obtailmd for B invertible and the lincar study 
of Serre and Zumbrun obtained for a pure Dirichlet's boundary condition. 

1. INTRODUCTION 
We consider a one-dimensional systenl of conservation laws with a small param- 
eter « set in the dornain x > 0. 
(1) u t + F(uS): = e(B(uS)v)x, x > O. t > O. 
where u e G n and F  H  n, B  H  nxn. ç will assume that F and B 
are smooth (Cm). We add to this system an initial condition us(0, x) = u0(x) and 
a boundary condition that we will detail later.  sume that the eigenvalues of 
B have nonnegative real part and that the tank of B does not depend on u. Ve 
will denote it by r, 1  r  n. Note that B is not necessarily invertible. We are 
interested in the limit of u e when « tends to zero. çX expect that u e tends to a 
solution of the inviscid problem: 
(2) ., + F() = 0 
with sonle bomldary conditions to be determined. At first we make the natural 
sunlptions to ensure the well-posedness of the Cauchy problem for (1) [6]. There 
exists a change of variable u  v(u) with inverse u = 9(v) in which the systenl can 
be rewritten as 
(3) g(v)t + f(v) = e(b(v)Vx) 
with the following properties: 
 (H1) b(v) is block diagonal, 
b(v) = 0 bi(v) " 
with b (v) e GL(). 
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 (H2) dg(v) is lower block diagonal. 

dg(v)= ( v O) 
(,) " 

with t)(v) E GL,-(). 
Bv analogy to the terminology in gas dvnamics, we shall refer to t' as the primitive 
variable. 
Next we assmne that (1) is symmetrizable mixed hypert»olic-parabolic: 
 (H3) there exists a positive definite symmetric (u) such that 
(1) E(u)dF(u) is symmetric, 
(oe) (,)(.)x. x  l()Xl =, x  , 
where  > 0. mM - stmMs for the scalar product of . 
k2, denote by v = (w, z) the corresponding block deeomposition of ç. 
Note that since [(v) =dF(u)dg(v), b( v)= B(u)dg(), setting S()=dg(v)t(v), 
we get that (H3) is eqnivalent to 
 (H3') There exists S(v) such thm, 
(1) S(ç)dg(v) is positive definite svmmetric, 
(2) S(v)df(v) is svmmetric, 
(3) S(,)b(,)x. x > .ll , vx = e 
$$% point out that (tt3)(1) implies that the inviscid svstem (2) is hyperbolic. 3Io» 
over (H3)(2) in,plies that dgtd9 is i,h,ck diagonal (see [11], Lemma. 4.1). Hence 
thanks to (H2), ce get 
 S(v) 
where S(v) is positive definite symlnetric. Consequently, writing the block de- 
composition of df as 

we get flore (H3')(2) that Sw(v)h(v) is symmetric. This means that the system 
obtained from (3) by removing the second equation is symmetric-hyperbolic. 
Finallv. we also assume that the hyperbolic and parabolic modes do couple: 
 (H4) The kernel of B does hot contain anv eigenvector of dF. 
The structural hypotheses (H1-H4) are verified by many physical equations as 
those of compressible gas dvnamics and nmgnetohydrodynamics. 
Next we make hypotheses to deal with the initial boundary vahle problem. We 
focus on the case of a noncharacteristic boundarv. We assmne that the boundarv 
is noncharacteristic for both the viscous (1) and the inviscid (2) systems: 
 (H5) dF(u) and ](ç) are nollsingular. 
Note that an inflow or outflow boundary condition nmkes the boundary noncharac- 
teristic in most cases for the Euler and Navier-Stokes equations. These boundarv 
conditions have a physical meaning since they appear iii problelns with aperture. 
such as in oil recoverv. The analysis of an impermeable boundary xvould be different 
since in this case the boundary is characteristic. 
We denote by q the number of eigenvalues of positive rem part of dF(U), and 
by p the nmnber of eigenvalues of positive real part of h(v). An initial boundary 
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value problem for (1) nee(ls p + r scalar indepen(lent bomdary conditions, and 
an ifitial boundary lue problem for (2) needs q independent scalar boundary 
conditions. We deal with bounda.ry conditions fl»r (3) that are linear with respect 
to the primitive variable v. çV write the bOulidary condition for (3) as 
(5) Lv(t'O)=(l'") (''O)=9.z 
where l is a linear mal» that has rank p and 9 is a given constant. 
In the following, in order to make energy estinlates, we assmne that the boundary 
condition (5) is "dissipative": 
*(H6) fl > 0. Vv  ç, VX = .¥ . such that Lv = 9- LA = 0. and wc 
have 
s(v)(v)x . x  -lXl . 
There are physical boundary conditions in the form (5) that satisfv (ti6). The 
case of the isentropic gas dvnamics will bc stud;ed Iwh»w. 
Note that thanks to hyl,otheses (H I-Ha), we have p ç q  p + r ([12]. Corollary 
1). Hence in the case q < p + r. there is a h»ss of bOmldarv condition when  tends 
to zero. It is due to a fast change of u  in a vicinitv of the boundarv: the b«nmdarv 
laver. In the noncharacteristic cas« the size of tlw bomdary laver is ¢. When ¢ 
tends to zero. the expected t,ehaviour of ,.« is ([2], [1], [12]) 
  ,.it(t. z) + U(t..r 

where u it is a soluti(,n of (2) with the initial condition ,t(0..) = u0(z) and some 
boundary conditions that we have to deternfine. U(t, z) is a boundary layer; it is a 
solution of a differential problem where the time is onlv a parameter: 
(6) (U)U' = (U + ,,'(t. 0))- F("(t.O)). 
U(t, +OE) = o. 
L,(U(t.O) + ,,"'(t. 0)) = g. 
Note that when r < , we have an algebraic differential system. This problem has 
solutions if and onlv if uit(t. 0) belongs to the subset C. where 
c = u+  ", u, u(+oc) = o. 
L,(U(O)+r+)=g. 
This sct C is called the set of residual boundary conditions. It was studied in the 
case r = n in [2], [4] and ill the gelleral Ça,S(? iii [11]. Assuming that u0(0) satisfies 
the boundary condition (5), i.e., 
(7) L(o(0)) = 0. 
we have 0(0) C (the associated profile ofthe boundary layer is U = 0). Moreover, 
thanks to (H1-H6) we can use [11], Lemma 4.2 and Theorem 1.1. C is a smooth 
submanifold in the vicinity of u0(0) that bas dimension q and that is transverse 
to the unstable subspace of dF(u0(0)). Consequentl3; thanks to a theorem of [8], 
there exist.s a continuous solution of (2) with the boundary condition uit(t, O)  C 
defined on [0, T] for some positive small time T. Assunfing some higher-order 
compatibilities between "u0(0) and C, we can even get a smooth solution u. Usiug 
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the smne method as in [4], we can show the existence of an approximate solution 
of (1) in the fornl 

(8) 

such that 

where 

^1 
i=1 

Lv(u«vP(t.O)) = O. uaPv(O.x) = uo(C). 
0,  + OF(, ) - eOx(B()Ox) = R  

]]R¢E]IL[O,TI,L 2 __ Cg M. 
Our aire is to show that the truc solution a f is close to the approximate solution if 
the boundary laver is sufficiently weak. More precisely, 
Theorem 1 (Nonlinear stability). Assumin9 (H1-H6) and that uo 6 Hr (+ ), there 
eists  > 0 such that if 
(9) sup (lO:U(t,')l + zlO:U(t,z)ldz ]O=U(t,z)ldz  a, 
tG[0.T] 
then 

tf e -- ri int --* 0 

(o) 
with 
() 

((()«f(v))  ÷ S()b(@X. X _ OlXl , 

VX  IR ' 

k(v)dg(v) skew-synmmtric. 

when ¢ -- 0 in L([O,T],L2). 
To prove this theorem, we actually need to start from a very accurate approx- 
imate solution -u app. Indeed, we will take M = 3 in the expansion (8). The con- 
struction of such a high-order expansion requires a lot of regularity on u i't (sec 
[4]). This is why we have to assume so much regularity on u0.  actually get a 
more precise estinmte: 
I1(. « -- .«)(t, ")11= + «110=( « -- )(t, ")11 + «11( "« -- )(t, .) Il  w,.  C«- 
Our method can also provide estinmtes in L([0, T], H ) for anv s. 
The proof of Theorem 1 relies on energy estimates. We use the primitive variable 
v; hence we work on the form (3) of the equation. X combine the energy estimate 
of the totally parabolic case (r = n) [2], [4] with an energy estimate of Kawhima's 
type [6] and a careful study of the boundary values. A key argument of the proof 
is the following lemma of [la]: 
Lemma 2 (S-K [13]). Ass.uming (H1-H4), there eists a skew-sBmmetric Iç(.u) and 
a positive constant 0 such that 
((Iç(u)dF(u)) Æ + (u)B(u))X. X  OlXl =, vx  , v..  u 
'(KdF+(K«F) t) 
where (KdF) Æ =  . 
Note t, hat setting k(v) = dg(v) tK(9(v)) we ean rewrite this result  
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Iu [9], Lenmm 2 aud the estimates of [6] combiued with pointwise Green's lune- 
tions bouuds were Mready used to prove the nonlinear asymptotic stability of weak 
time-iudependent viscous shock profiles for (1). The asylnptotic stability of a time- 
indepeude,t profile of the boundary laver together with the stability of other non- 
line waves was studied in [10] for the isentropic gas dvnamics rewritteu as a p 
system in Lagrangiau coordiuates. 
Let us give an example of an application of our theorem. C, ousider the isentropic 
gas dvuamics where v : (p, v), p being he mass deusity and v the fluid velocity, 
pv ' pv  + p(p) ' o (p) " 
Here we assume that u > 0 and that p' > 0 (hyperbolicity). The sound speed is 
c(p) = . (H1-H4) are verified: moreover, the eigenvalues of dF are v  c and 
the eigenvalue of (the 1 x 1 matrix) h is v. 
Let us first consider an outflow bouudarv condition 

(12) 
(13) 
(14) 
where 

(t. 0) = v- 
with v- < 0. In this ca, l = 0, mM (H6) I)ccomes 
v-}Xl 2 <-lXI2; 
hence, it is satisfied. The compatibility condition (7) becomes vo(0) = v_. It 
suffices to impose Vo((I) + c(po(0))  0 to get (H5). 
If we consider an outflow boundary condition 
v(t,0)=v_, p(t,0)= 
where v_ > 0, we bave 1 = Id, b:er i = {0}, and hence (H6) is true. The com- 
patibility condition bccomes po(0) = p-, vo(0) = v_ and hence we get (HT) if 
v_ -c(p_)  O. Moreover, in the case v- -c(p_) > 0, we have q = p+r = 2; hence 
there is no boundary layer and the hypothesis (9) is always satisfied. 
For a more general discussion of the various boundary conditions for the non- 
isentropic gas dynamics, we refer to [12]. 
As in the totally parabolic case r = n, the smalluess assumption (9) in Theorem 
1 is linked with the stability of the bomMary laver. In [12] an example of a large 
unstable boundary la.ver is given. To understand the mechanism of instability in 
the boundarv laver, we set 0 = 
 . , z = , we fix some time T in app and we linearize 
about the leading terre of u app with respect to . We get the linear svstem 
Oou = 
Ldv(u(r, O) + U(T, 0))v(0, 0) = 0, 
(0, z) = o(z) 

£-u = (b(ut(T,O) + U(T,Z))U' + dB(ut(T,O) + U(T,Z))uU'(T,Z) 
- d((,0) + U(-,z)) . 
Here ' stands for 7" We will say that the profile of the boundary layer u it (v, O) + 
U('r, z) for some fixed "r is linearly stable if t.he solutions of this system tend to zero 
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when t tends to +o« The linem" stability is linked with the spectral stability as 
was show,, in [15], [9]. [141. Let us define the domain of/2,- as 

/9(£-)= u=dg(l/(T,z))v, v= . w e HI(]+), z e H2(]+), Lv(t.0)=0 
z 

where V(r, z)is defined by g(V(T,Z))=iWt(T.O)+ U(r, z). 
In [12], it is shown that the essential spectrum of /2- is confined in {7.eA < 
(I} U {0} thanks t.o (H1-H5). In the unstable half-plane {7.eA _> 0}\{0} the spec- 
trum only consists of eigenvalues. Conseqn('ntly, a necessary condition for the linear 
stabilitv of the boundary laycr is that the operator/2- does hot have eigenvalues in 
the unstable half-plalm {7.e A >_ 0}(spectral stability). An Evmls flmction machin- 
ery was developed in [12] to find sufficient conditions of instability. 
In the first part, we show that spectral st.ai)ility holds for weak boundarv lavers. 

Theoreln 3 (Spectral stability). There exists 5 > 0 such that, assuming (HI-H6) 

(15) 

thon F_«. docs hot bave ei9cvalues i the unstable hall-plane {R.c A _> 0}. 

The proof also relies on energy estimates. We first give a direct proof of Theorem 
3 because it seems more enlightening to present the main ingredients of the proof 
of Theorem 1 in the simpler linear time-independent setting of Theorem 3. This 
result is not used in the proof of Theorem 1. The result of Theorem 3 could be 
deduced from direct energy estimates on the rime evolutionarv svtenl (12), (13), 
(14). Nevertheless it is interesting to study the spectral stability since ve can expect 
that, as in the totally parabolic case, the sharp assmnption of spectral stability 
implies the nonlinear convergence result [5]. 
Note that out result of Theoreln 3 (obtained bv a different method) implies 
the result of the appendix of [12] where only Dirichlet's boundary conditions were 
considered for (1). 
In the second part, we give the proof of the full nonlinear stabilitv result of 
Theoreln 1. 

2. SPECTRAL STABILITY 
In this section, we prove Theorem 3. We studv the eigenvalue problem 
(16) Au- £«u = O. 
(17) Ldv(uit(T.O) + U(r, 0))u(0) = 0. 
Setting Ut(T,O)+ U(T,Z) = g(I'),  = dg(I')v (ve onfit the dependence with 
respect to r in this section since r is fixed), we rewrite the probleln in the prinfitive 
variable. Hence we have to study the equation 
(18) AA°v + Ae' - (bv')' = A'v + (Cv)', 
(19) Lv(0) = (lw(0) ) 
:(0) =0 
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where v = , An(z) = dg(I'), A(z) = df(l'), b(z) = b(l'), and Ch 
z 
db(V)hI a. Note that we have the estimates 
for some M > O. where S(z) stands tBr 
Moreover, note that thauks to (H 1), 
(21) II(Cv)' = O. 
where Hw = w. 
Z 
Let us assmne that there exists a nonzero soluti.n of (18), (19); withom 1.ss of 
generality, we assume that 
(22) IIvll = 1. 
In this section, since we dcal with flmctions that take complex values, we denote 
bv u  v the scalar product of C n, 
and by - the associated uorm.  th«u define 
II,ll 2 = -0., (.) = ,«()-,,(.)dz. 
 split the proof of the theorem into several lemmas. We will collect, all the 
estimates at the end of the section to reach our conclusiou. 
 first give an energy estimate iu the saine spirit as in the totally parabolic 
case [2], [4] or in the pure Dirichlet's bouudary coudition case [12]: 
Lemma 4. Assm,e that v is a solution 4 (18). (19) that satisfies (22). Then. 
when  is scieztlg small, we have the estimate 
(z3) n  + l'll  + l,(0)l  5 c. . 
(24) «  + 11'112 + lw(0)l =  cs(I,.(0)l = + I1'11=), 
(25) IZ, 1  c( + I1"11). 
Note th.at the first estiçmte (23) 9ives 
(26) e   C3. 
Proof.  first use the saine energy estimate as in the strict]v parabolic cse [2], 
[4] and the full Dirichlet case [12].  take the Hermitian product of (16) bv 
(in this section, we wil] denote S(V) by S for the sake of simplicity) and we take 
the real part, getting 
« (SA%, v) + e (SAc', v) - « (S(bv')', v) = e (SA'v. v) + Re (S(Cr)', v). 
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Since SA is syminetric, we get 
Tae (SA-d, v) = 

> 
thanks to (tt6). Note that 
I(SA)'v,v)l <_ C + 
Next, integrating by parts, we bave 

1SAv(O).v(O) 
 ( (SA)'v, ,,) - - 
-((SA)'., ,) q- cl,(o)l 2 

IV'llvl 2 

Tac ( s(bv)', v) = -R « ( Sbd, ,') - n« ( ( S'bv', ,) - Tac Sb,' (0). v(O). 

Thanks to (H3'), we have 

nloreover, we bave 

« (Sb',') _> zlz'12; 

Tae Sbd(O) . v(O) = 0 

thanks to the structure of the matrix b given by (H1) and (19). Using again (ttl) 
and (4), we have 
( 1 0+° ) 
(('bd,.)) = (<b,z',z) < C ollz'll 2 +  IV'llzl 2 
for every q > 0 by using the Young inequality. Moreover, we have 
[(A'v, v)l _< C IV'I Ivl 2 dz 

and 

(27) I(S(o)',,)l = 
J0 +oe lJ0+°c ) 
< c IV'l MIz'l < C@ll'll 2 + IV'lll  
thanks to (H1) and (20). 
Çollecting these various inequalities, we have shown 
«ll.vl12+llz'l12+l,(o)l 2  Cllz'll2+C() IV'llzl2+C() IV'lM 2. 
To conclude, we first choose  = , then we use z(0) = 0 through the inequality 
14x)l 
+ c[ + _ 
(28) IV'lll  < xl"lll»'ll  < Cllz'll  
and finalb; we absorb the terres Celle'Il  l,d Ç(e) f IV'III  i, th i prt 
11'11  f   -me,tb" n. hi proe (23). 
To get (24), we use 
(29) 
IV'll,l   olV'lll'll  + I(O)1  IV'l  O(l(o)l  + II,'ll). 
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To prove (25), we also take the scalar product of (16) by St,, we take the imagi- 
uary part and we onlv use 
z,,, (s,,.", ) : -z ( Sb,', ') - Z,,, ( ( Sb)',,', ,,)  C(llz'll  + I1'111111)- 
 get 
z, llll  5 C(ll,'ll  + I1'111111 + allll)  
To conchlde, it suites to use (23), whi«h gives, in pa.rticular, 
I1'11   Call,,ll  
and the normalization assmnption (22). 
In the case of a pure Diri«hlet boundary condition, a weighted energy estimate 
on the hyperbolic part of the system (that is to say on the first n - r equations) 
w used in [12] to bound the terre 
This estinmtç was similar to the one used by (ioodman [3] for tire stabilitv of viscolls 
shock profiles. This was ecient because of thc upwind propagation. In out more 
general seting we use an energy estimato of "Kawashima's typo" [6], [9]. 
Lennna g. Assume lha¢ v is a solvtion of (18), (1.9) tat sat.sfies (22). Ten for 
scieal small . we have 
(30) II.'ll 
Proof. We use the nmtrix k givcn by (10).  apply b to (16), we take the scalar 
product by v' and we take the real part. Using "e(kAv', v') = ((kA)*v ', v'), we get 
Here we have used the estimates (28) and (29). 
Using that kA ° is skew-Hernlitian, we have (kA°t ', v') E R since 
(a-A°v, v') : kA°v(O), v(O) - ((A°)'v, v) - (kA°, ', v) = -(A".'. v). 
Consequently, we have 
I«(a(A%, ¢))1 : Id )(.4 , ¢)1 S C«allll I1¢11. 
Siuce we have the estimate (10) 
((A)''. ')  011'11  - ClI'II , 

we get 

IIv'll 2 < C(llz'll 2 + IIz"ll IIv'll + ne  I11111'11 + all'll 2 + al'(o)12). 
and hence choosing r/> 0 sufficiently small, using the Young inequality and (9) we 
bave 

/ 
(31) II'll 2 < c(,1)l, llz'll 2 
+ 
Consequently (30) is proved. 

(ne x)211,112 +- I1»"112 + al,(o)12). 
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To end the proof of the theorem we would want to estimate []z"[[ with respect 
Lemma 6..4.ssne that  is  solution of (18), (19) that sati.sfies (22). Then. 
when 6 i,s stcientl sntall , u,e bave 
I1"11 
(32) 
Pçoof.  take the derivative of (16), getting the equation 
",' + A,," - (b,'")' = O(IV'l)(l I + I"1 + I'1) + (C)". 
Thc proo[ is very similar to the proof of (23), in that we take the scalar product of 
t.he equati(m hy Sv  and we do an integration by parts. The «boundary'" terres do 
hot vanish since d(0) does uot satis" the bOulMary condition (19). b just point 
mt that to lmund the terln ((S(Cv)",v') we also do 11 integratiou bv parts as iu 
(27) to get an estilnate independ¢,lt of 
2.1. Proof of Theorem 3. ke now give the proof of Theorem 3. To conclude. 
we first have to eliminate z"(0) and v(0) in (32). 
We first express w'(O), lhanks to thc hyperboli« part of equation (18): 
,.'+ .a, ,' + .a.=' = O(1"1)(1 I + I'l). 
where A = A. .4 . Note that we make a crucial use of (21). 
Since the boundarv is non«hara«teristic for the viscous svstem. A is nonsingular: 
moreover, thanks to (2g), (25), we have 

(33) 
We dedu«e 

I,'(o)12 _< c'(l,(o)l 2 + I»'(o)12 + 1,(o)1211v'll 2) 
(34) <_ C(l,,(0)l 2 + I-'(0)12 + llv'll 2) 
since thanks to (2g). we have w(0)]   
The next step is to estimate lz'(O). We use the classical Sobolev inequality 
(35) Iz'(o)l oe  211»"1111zll  ,#llz"ll oe + llzll oe 
for every q ) 0. Hence it SllCeS to estimate z"(0) in (32). %7 use the parabolic 
part of the equation 
4 ' 
O(v) + Aa,«' +. : -b," = O(IV'I)(II + Idl)+ 0(1111). 
We get. thanks to (23), (25), (34), and (35), 
1 12 11,,112) 
(o) I"(o)12  c'(l(o)l 2 + ( + ,])11'1 + 11'112 + 
Next. we choose q such that C'q < 1, and we replace (34). (35), (36) in (39]. getting 
(7) I1"112  ç(lldll 2 + I1'112 + 1,,(o)12). 
çinally. «oll«ting (23), (30) and (37), w bave shown that 
(« )( - raz) + ( - c')ll«ll  + ( - C)l.,,.(o)l 2  o. 
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Hence if 5 is sufficiently small, this gives if T¢e A > 0, z = 0 and w(0) = 0. The 
hyperbolic part of the equati(m then t)e«omes a first-older ordinarv difl,rential 
equation involving only w: 
,,' : (A)-(-X, ' + 0(1"1),') 
with the boundmT condition w(O) = 0. Conseuently we a.lso get w : 0. This ends 
the proof of Theorem 3. 

3. NONLINEAR STABILITh 
In this section we prove Theorem 1. We use the form (3) of the svstcm. Setting 
u e = g(v e) and u app = g(vaPP), we have the two systcms 
J(Ve)t ÷ (f(ve)x = 
Lve(t. 0) = g. 
,'e(o. *) = ,'o(.r) 

and 

Setting v E = v app + v (we Olllit thc dependence of t, iii ), We rcwritc Olll" lrollt'lll 
s 
A°i)tv + Ac%v - eO.(bO:v) = R G + 310 + ]111 + 

(38) 
where 

A ° 
b = 
M ° = 

AI 1 = 
11 '2 = 
Note that v satisfies the 
(39) 
and the initial condition 
(40) 

L,(t.O) =0 

(41) 
where 

v(O. x) = O. 
We «hoose C suffi«iently large such that 
Q <_ Ce N, 
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for solne large N which will be chosen later. 
To plove Theorem 1. we lise the classical çontinuous induction argmnent ([3], 
[7], [51, [9]). Let IlS define 
+ «Oz(s)l[ = + «ll0,z(s) = d. 
+ t,(s. 0)l = + «=10¢,(.. 0)1 = + «410,¢,,(t. 0)1= d.. 
Note that thanks to (39), (40), (4), we have 
(42) E(o)  ç'e . 
Using the classical short-lime theory, we define 
r* = ,,p{ç« e [0. ç],   oh,ti,,, of (3S), (39), (a0) 
e [0. r). E(t)  e ' } 
Sll(.h 
that 
Vt 
where we choose Ni < N. There are wo possibilities: 
() T* = T, 
(2) T* < T. and E(T*) = e N. 
In the following, we show bv an energy estimate that we cannot be in the second 
case. This will show Theoreln 1. 
Let us define a()  
Z = Slip slip 00îI" t,-- . 
a ê te[0.T] 2>a>l,2kk0' 
Al first we need an elementary lemlna about the estilnates of the nonlinear quan- 

tities that arise in (38). 
Lemlna 7. Vi = 0.1. 

IOAI < c(10«vl)(1 -4-10,vl), 
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/ 
Remark 8. 
  '--t x '* Z 
 also point out that, actually, we will hot usç the case o = 2,/3 = l in Lelnlna 
7. 
ç now corne to tlw proof of oto nmin theorem l. In the proof. C stands for a 
number that is independent of ¢ but nmy dOl»end on T. 
Since bv classical Sobolov emtmddings, we have 
T* (T* 
1Ozle 
and since by using the hyperbolic part o[ efluation (38) (thal is o say, the w 
component), the nonchmacteristic assmnption and Remark 8. we bave 
we gel the estimale 
(43) 
Ce,- 
Çonsequcntly, we choose N  6 and 5 < N < N. This allows us to use Lemma 7. 
Moreover, thanks to (43), we oi)tain })y contimfity ri'oto (H6) that 
(44) S,,,(v 
since 
Note that our smalhmss assmnption (9) and (6) imply that 
(45) p (z)+ () d: + () dz 5 C. 
z6N+ , , 
As for the spectral stability, there are four steps in the proof. Ve first lnake 
the energy estilnate of the totally parabolic case; next we lnake an estimate of 
Kawashima's type and an estimate on the space derivative of equation (38). The 
final step is to estimate the boundary values. For this, we replace (25) in the time 
evolutionary setting bv ai1 energy estilnate on the time derivative of the equation. 
At first, let us make the saine energy estilnate as for the totallv parabolic case 
[2], [4]. 
- 

(46) 
,0 ÷°c 

a(ç)lvll0xvl _< CllOvll a(-)lvl 
_< c(«ll0.ll 
we easily get., after absorbing the terres CS]w(t. 0)12 by the ter,,, ct[w(t, 0)12 
left-hand side, 
(48) Ot(SA°v,v)+a[u'(t,O)[ 2 
where S stands for S(1 app ÷ ,). 

on the 
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Note tlmt an estinm.te such as (47) is needed to bound the teInls 
(eSi:).(bOxv), z,), 
Next we replace the estinmte (25) of the spectral sta.bility bv an estinmte on the 
time derivative Ov. Since 0«¢, still satisfies the bomdary condition LO«v = 0. we 
can perform ¢he saine computation as previously on the rime derivative of (38). 
Thanks to Lemnm 7. we get 
(4) o,(sA%.o,)+lO,,,(t.o)l=+5dlO,.ll = 
the ideas of the computation have been use& 

We do hot give more details since ail 
\Ve just point out that we bave used 

(50) 
and that fo bound the terre 

we perform an integration by parts and use the block assunlption (H1) and (4). 
This ternl is then dominated bv 
C(ll,ll = ÷ 11,=1 + qllo,=ll =) 
for every q > O. , absorb the last factor by the terre flllO,.=ll  . the left-hand 
side v choosing 1 suciently small. 
Note that for the moment, we do hot use an ineqlmlity similar to (46) to bound 
terlns Sllch as 
+oe a()[tt' 2 
 bomd this terre by expressing Ott', thanks to equation (38) and by using esti- 
mates such as (46) and (47). Then we get 

COIlleS frolll 

(Ç ' ) 
_< c I,.(¢.0)1 =÷-IIôx¢,ll = . 
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Replacing (51) in (49) yields 
(5"2.) ,(S'A",,,.O,,,) ÷ lO,.,(t.o)l 
I 
 c( « + I111 = + Ila,,ll = + (] + )lla=l +   
As for the spectra} stability, thç next step is to use the "Kawashima'" estimate 
[6], [9]. b aI,ply k(v 
(53) (kA°0,,..O,v) +(('- "' . - 
 use the cruciM estimate (10). This 

Next we xvrite for every q > O, 

= I«(»O,.bO,.,..O,v) + 
< ('((« + )110.,'112 + llO,zlli.,,ll) 
C((e + d + )110,11 = + 
where here we bave used the blo«k structure assmnt)Iion (l[ l) and tlw 5omig in- 
equality. Using Lemma 7, the ung inequality, and (47) we bave 
(55) 
Bv the smne method, we ge simi}ar estimates for (kM °, O.u) and (kM 2. Ozt'). To 
handle (kA°&u. 0«v), we write 
Performing an integration bv parts in the lt factor al»ove, we get 
+ (¢(kA°)t,.Otv) + 
and hence, since kA ° is skew-symmetric, 
1 
(O,(kA°v.Ov) - (Ot(k.4°)v.O«v) 
(kA°Otv. Ov) = 
+ kA°(t. 0),9.0). O,,,(t. O) + (O(.A°),,, 0,.,.)). 
To bound (O«(kA°)v. O,v). we uoe 

- ( )MIO, vl d« 

thanks to (46) and (51). 
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This yields 
1 o 
(56) (kA°Otv. O«v)=- 
for any q > 0 by use of the ung inequality. Consequently. collecting 
(56), choosing q sufficiently small and using our assumption (9) to absorb (e + 
)ll0vll  in the left-hand side if  is sufficiently small, we get from (53), 
+ 
In conchlsion, it relnains to estilnate 
with respect to z and we perform an energy estimate similar to (2), but now the 
boundary terres in the integration by parts do hOt vanish. Using Lemma 7 and 
estimates such as (46), (47), and (50), we get 
(58) O,(SA"Ov,Ov) + llOll  
 c  + (1 + 
+ Iou(¢. 0)1  + 10(, 0)1). 
Note that to estimate the terre (Ox12,âxv) we have perfomed an integration 
1)y parts to avoid that terres involving IlOfll appear and that to estimate 
(SOAaOcv, Ov), we write 
f+ 
I(SOA°O,,Oæ)I 
and we use (51). 
As for the spectral stability, the next step is to estimate the boundarv values 
Ov(t,O) a,,d 0(.0).  rt write th ,og, of (aS), 
1 
(59) 1o.(t. 0)l 2 
for some q sufficiently smal] so tiret the terre 
wi]l be absorbed by g]e ]efg-hand side of (58). 
To est.image Ow(t, 0), we use the hyperbolic parg of the equation, the facg that 
the boundary is noncharacteristic and Lemma 7, getting the estimage 
IOw(t,0)l   c(IR«(t,0)ff 
(60)  c(l(t, 0)1 
1 
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To estimate O.zz(l, 0), we use the parabolic part of the equatioll. This vields 
(6a) 2lo=z(t.o)12  C(l(t.o)l 2+lo,.,(t,o)l =+«,llo.=zll  
+ --0»1  + ( +  , - 
Next, putting (59), (60), (61) in (58) gives the bound 
(62) Ot(SA°Ov,Oz ,) + «]]O=zll 2 
(    ,,,,,, 
 c 0  + (a + )1111 = + (a + )110.11 = +  . 
Fina.lly, we «onsider (48) + «2(52) + «(57) + F«2(62) with F > 0 suffi«ientlv large 
and independent of . lntegrating ri'oto 0 to t, we get the estimate 
(63) IIz,(t)ll 2 + 2110z'(t)ll2 + 2ll0=(t)ll 2 
+ Ç «ll0z,(s)ll 2 + «31lOtz(s)ll 2 + l'«3lloz(s)ll 2 d.) 
 ç IIz,()ll 2 + «2llO,,,()ll 2 d. + e N) 
thanks t.o (dl), (2), and (9). Note that we havc also used that N'.4" is positive 
definitc symmetri«. Thanks to 5ung's inequality, we bave 
for any q > O. Hen«e «hoosing q su«iently snmll su«h that Çq < 1 and then F 
C 
sucient}v large su«h that ç > , we finallv get 

«allO, xvll = + alO,.,(t, o)1 =. 
Next using Lemma 7. we perform estimates analogous to (49), (57), and (62} for 
Ottv, Ot«v and Ot«,z respeetively. 
Finally. we obtain the estimat.e 
E(t) <_ c(N + fotE(s)d.s). 
Therefore, by Gronwall's Lemma we bave 
:(t) < c«  vt  [o.w*]. 
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Hence E(T*) < ¢N since N > Ni and hence T* = T. 
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HOMOLOGY OF PSEUDODIFFERENTIAL OPERATORS 
ON MANIFOLDS WITH FIBERED CUSPS 

ROBERT LAUTER AND SERGIU MOROIANU 

ABSTRACT. The Hochschild homology of the algebra of pseudodifferential op- 
erators on a manifold with fibered cusps, introduced by Mazzeo and Mel- 
rose, is studied and computed using the approach of Brylinski and Getzler. 
One of the main t.echnical tools is a new convergence criterion for tri-filtered 
hall-plane spectral sequences. Using trace-like flmctionals that generate the 0- 
dimensional Hochschild cohomology groups, the indcx of a fldly elliptic fibered 
cusp operator is expressed as t.he sure of a local contribution of Atiyah-Singer 
type and a global terre on the boundary. We announce a result relating this 
boundary terre to the adiabatic limit of the eta invariant in a particular case. 

1. INTRODUCTION 

Let X be a compact manifold whose boundary OX is the total space of a locally 
trivial fibration  : OX -- " of closed manifolds. In contrast to the case of closed 
manifolds, a great number of algebras of pseudodifferential operators tan lmturally 
be associated to this geometric situation. To list a few of them, recall that Richard 
Melrose introduced a concept of geometric micro-localization that associates to 
certaiu classes of Lie algebras V of vector fiel(ts on X algebras of pseudodifferential 
operators; the Lie algebra /' is then also called a boundary fibration structure [26], 
[30]. The corresponding pseudodifferential calculus koV(X) contains the universal 
enveloping algebra DiffV(X ) of /', the /'-differential operators, as a subalgebra, and 
V-elliptic (pseudo)differential operators (a notion that has a uatural neaning in 
the context of boundary fibration structures) can be inverted withiu v0V(X ) up to 
operators of order -oc. Possible candidates for boundary fibration structures on 
our manifold X are, for instance, 

(1) 

:= {V G C(X, TX)  V is 
tangent to the fibers of rr at OX } 
where ç  X + OE+ is a defining function for 0X, i.e., 0X = {ON = 0} and 
does not vanish on OX. Note that the fibered cusp structure ç(X) depends slightly 
on the choice of the boundary defining functiou. This depeudence is discussed in 

(edge-st n ct ure), 
(double-edge structure), 
(&fold-edge structure), 
(fibered-cusp structure). 

Received by the editors July 15, 2002 and, in revised form, January 16, 2003. 
2000 Mathematics Subject Classification. Primary 58J42, 58J20. 
Moroianu was partially supported by a DFG-grant (436-RUM 17/7/01) and by the European 
Commission RTN HPRN-CT-1999-00118 Geometric Analysis. 

(2003 American Mathematical Society 

3009 



3010 ROBERT LAUTER AND SERGIU MOROIANU 

more detail in [23]. Corresponding pseudodifferential calculi were constructed and 
studied in (22] (edge-structure), [17], [32] (double-edge structure) and [23], [31] 
(fibered-cusp structure). A pseudodifferential calculus for the ¢-fold edge structure 
has been considered so far only for the very special case 7r -- id - OX -- - and 
t = 2. under the nanle quadratic scattering structure [46]. The reader can easil? 
invent more Lie algebras that might introduce boundary fibration structures, but 
one should be warned that it is not at all straightforward (and sometimes even 
impossible) to establish the additional properties that ensure the existence of a 
pseudodifferential calculus. For more details, we refer to [26], [32] and the forth- 
coming book [25]. 
It is worth noting that bv integrating appropriate Lie algebroids as Nistor did 
in (39] and using general groupoid techniques. (slightly smaller) pseudodifferential 
calculi associated to many interesting boundary fibration structures can be con- 
tructed simultaneously. We refer to the survey [18] for many examples of this 
construction. 
To give an idea of how the different boundary fibration structures in (1) look 
locallv, we use coordinates (x, y. z) E II+ xlI xlI in a local product decomposition 
near the boundary, where x is the restriction of the boundar3 defining function 0v. 
y is a set of variables on the base  lifted through :. and -- are variables in the 
fiber. Then anv vector field in one of the boundarv fibration structures from (1) 
can be written locallv as a linear combination over C(\') of the following basic 

vector fields: 

.2) 

(edge-structure), 
(double-edge structure), 
(Lfold-edge structure). 
(fibered-cusp structure). 

In the present paper we compute the Hochschild homolo- and study its relation 
to index problems for the fibered cusp calculus, i.e., the pseudodifferential calculus 
associated to the fibered cusp structure. Starting with the work of Wodzicki [43]. 
.441 [45] and B13linski and Getzler [3]. [4] on Hochschild and cvclic homolo" of 
[pseudo)differential operators and on non-commutative residues and Euler classes. 
homology of pseudodifferential operators has attracted much attention, hOt onlv 
because of it relation to index problems as explained for instance in [34] or [15]. 
.16]. but also because of its connections to the non-commutative geometry of Alain 
Connes [6]. Indeed, algebras of pseudodifferential operators are non-commutative 
algebras naturall3 associated to (sintlar) geonletric situations (boundary fibration 
tructures), and there are good reasons to expect that the study of invariants of 
these non-conmutative algebras xill reveal information about the geometry and 
help us to understand what a non-commutative manifold (possibly with sintlari- 
ties) is supposed to be [7i. 
The interest in the fibered cusp calculus originally introduced in [23], [31] has 
groxm considerablv since Nye and Singer [38] nsed this pseudodifferential calculus 
to prove an L2-index theorem for Dirac operators on S  × 3, and Vaillant [42  
applied the fibered cusp calculus to study spectral and index theory for Dtrac 
operators on manifolds with generalized fibered cusps that occur for instance when 
compactif3ing tank one locally symmetric spaces [36]. 
From (2), we see that the fibered cusp structure interpolates between two in- 
teresting boundary fibration structures on manifolds with boundarv namelv the 
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scattermg structure corresponding to the trivial fibration 
locally given bv x'20x and .r0y, and thc cusp structure corresponding to ç : OX -- 
I" = {pt} locally gencratcd by x'20« and 0y. For our purposes it is interesting to 
note that the scattering structure is also a special case of the doublc-edge structure 
with ç : OX -- 1" = {pt}. The Hochschild honlology for various algebras associated 
to the cusp algebra has bcen computcd in [34], whereas the Hochschild homology 
for the analogous algebras in the scattering setting can be extracted as spccial cases 
fronl the homology of the double-edge ca.lculus in [14]. We do hOt knoxv any way to 
combine these results to get the Hochschild homology for the fibered cusp calculus 
directly; nevertheless, the results mentioned above tan be used to double-check the 
computations of this pal»er. On the other hmld, when comparing the results for the 
double-edge calculus with those for the fibered cusp calculus, we see that Hochschild 
honlology groups yield a functional-mlalytic way of distinguishing between the two 
calculi. 
The fibered cusp calculus 

k,.rnZ 

is naturally a bifiltered algebra., with the first filtration (m) given by the sylnbolic 
order of the pseudodifferential operator whereas the second filtration (k) corre- 
sponds to the (negative of the) order of vanishing at the bomdary, more precisely 
at the q)-front face, a notion that is explained in Section 2. 
The algebra 'Z(x) has three interesting ideals, namely 

The residual ideal oe'-oe corresponds to operators with Schwartz kernels in the 
space of all smooth half-densities that vanish to infinite order at the boundary; it 
is easily seen to have Hochschild homology only in dinlension 0, where it is one- 
dimcnsional, the isomorphism to C being given by the usual operator trace [34]. 
We are hot going to stress this ideal anv further. Following [34], let us denote the 
following quotients bv 
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The foIIowing diagraln smnmarizes the situation, where the horizontal and vertical 
seqnences are exact: 

(3) 

- '(x) »  -- o 

\Ve are going to compute the Hochschild holnology of the ideals /2- and 
as well as of ihe quotients Jt,, Jt0, and 4Jt,. Moreover, we give geometric 
descriptions of the long exact sequences in Hochschild homology arising from the 
six short exact sequences (three rows and three cohllllllS) ill (3). Not surprisingly, 
for the computation of the homology of the symbolic algebras Z, ,A, and 
we use the saine approach as Brylinski and Getzler in [4], and obtain, for k E No 

* t* 
where Hre  (resp. t») stand for the relative (resp. absolute) de Rham cohomol- 
ogy of a COlnpact manifold with bouldary. The fibered cusp cosphere bundle 
shows up naturally in the analysis of the fibered cusp calculus and is in fact dif- 
feomorphic to the usual cosphere bundle S'X, though hot naturally. However, 
even though the idea of the COnlputation is the saine as in [4], namely to use an 
appropriate fornl of symplectic dualit.y to identify the d 1 ill the spectral sequence, 
convergence of the spectral sequence is an issue that is often neglected, and we 
spend some time to give the complete argunlent, hl fact, we are naturally led to 
consider tri-filtered differential complexes, where the filtration indices run in Z. In 
this setting convergence issues become very complicated, although it is rather sat- 
isfactory that they can be affirmatively answered. We give a general criterion for 
convergence which might well be applied in other situations. 
More surprising is the computation of the homology of t.he boundary idem 
and the boundary algebra ,Aô. We use a non-canonical morphism of algebras 
0  ggs--c,ct(Yé)  tPêoe(X) of the algebra of (compactly supported) scattering 
pseudodifferential operators of order -oo on the cylinder } := Y x [0, e) to the 
smoothing fibered cusp operators on X to reduce the computation of the Hochschild 
honlology of 4ï'0 to that of the snloothing boundary ideal 0 of the scattering 
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calculus on té. More precisely, we have 
HHk(qo)  H't+l-k(} -)  H't-k(}'), 
where, for simplicity, we have aSSUlned that the base t" of the fibration is orientable: 
the general case is obtained in Theorem 4.6 using cohonlology with coecients in 
the orientation bundle. The homology of the fibered cusp botmdary algebra follows 
then froln a long exact sequence that we describe explicitly iii geolnetric terres 
(Proposition 6.1, Theoreln 6.2). 
Starting from a differont approach using difforentiable groupoids, Bonalnenr and 
Nistor [2] developed a very general lnaehinery that computes, when specializing to 
our setting, the E  terln in the spectral sequence of the Hochschild homology of 
the symbolic algebra eM. Finally, note that computilg Ilochschild homology for 
algebras of pseudodifferential operators is hot restricted to the algebras nentioned 
above: for instance, the Hochschild holnology for Bouter de Monvel's algebra on 
a compact lnanifold with bomdary has been computed by Nest and Schrohe [gT]; 
the second author has COlnpUted the holnology for the adiabatic limit algebra [35]. 
The second part of the paper is dcvoted to residue flmctionals, traces and in- 
dex forlnul for the fibered cusI, calculus. As il [14] or [34], we introdu«« sev- 
eral linear flmctionals on 'g(X) that descend to traces on some of the ideals 
and quotients considered above by taking appropriate coecients in the Laurent 
expansion of the lneromorphic extension Zo,ç(,4 ) of the double-zeta fllnction 
(A,z)  (AOQ-); here, Q e è'°(x) is a fixed elliptic, synlmotric, strictly 
positive operator. Using these fitnctionals, we tan give a forumla for the index of 
Fredholm operators A in the fibered cusp calculus that extends the Ativah-Patodi- 
Singer formula for closed mallifolds in the form [g4] to the fibered cusp calculus: 
index(A) = AS(A) -li,n«ç(A)/2. 
Here AS(A) is given in terres of Hochschild homological fimctionals that depend 
only on the symbolic behavior of the opera.tor A but still involve an inverse of A 
up to trace class remainders, while liln« (A) is a terre that depends only on the 
behavior of A near the boundary. The terre AS(A) can be linked to the asymptotics 
of the heat kernel [13]. The notation lime (A) is motivated by the fact that in the 
special case t" = S  and A a differential operator with a Dirac-type decomposition 
near the boundary, the non-local boundary contribution to the index can be identi- 
fied as the adiabatic limit of the eta invariant of the "restriction to the boundary" 
of A. This result is developed in [13]. 
The paper is organized as follows: In Section 2 we review some basic facts about 
the fibered cusp calculus; Section 3 is devoted to the construction of the morphism 
O  .t(t)  (X), which is used in Section 4 to define a generalization 
of the Hochschild-Kostant-Rosenberg map in order to determine the Hochschild 
homology of o- The computation of the Hochschild homology of the symbol 
algebras eZ«, eA«, and eAo.« can be found in Section 5. The long exact sequence 
in Hochschild homolo arising from the boundary sequence 
is described iu Section 6 in geometric terres. In Section 7 we introduce and stndy 
various trace-like functionals on the fibered cusp calculus that generat.e the 0- 
dimensional Hochschild homology groups. As in [16], [14], [34], these functionals 
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tan be used to derive an in(iex fornmla for fully elliptic fii)ered cusp operators. The 
fornnlla is presented in Section 8. 
The I)a.per comains two appendices. In Apl)endix A, for the convenience of the 
reader, we review the definition of Schwartz (resp. sylnbolic) sections of vector 
bundles. Appendix B contains the definition of the Hochschild chain spaces for 
topologically filtered algebras in the sense of [2], and a general criterion for the 
convergence of tlle spectral sequence associated to a tri-filtered complex, which is 
applied in the body of the paper to the Hochschiht complexes of the synlbol and 
bomldary (I)-algebras. 
As usual, wc write s[r](V) f(»l" thc spa('e of ail Slnooth flmctions I \ {0} --, (2 of 
a vector lmndle t" that af(' positively homogelmous of degree m  C. 
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"2. [{EVIEW OF TIIE FIBEED CUSP CALCULUS 
I11 this section we rcvicw l)asic properties of the fibered cusp cak'ulus. For details 
and n,ost of the proofs we refer to [23], [31], and [42]. 
The bered ('usp structure spa«'e. Throughout this paper, X = X N stands for a 
smoolh, ('Oml)act manifol(l wilh bomdary OX that is the tot space of a locally 
trivial fibration ç  0X + " = yn of closed manifolds. For simplicity, let us 
assume that 0X is connected: so the fibers of ç are ail diffeomorphic, say to a 
closed manifold F = F TM. For the dimension N of X we thus have N = n + m + 1. 
Furtherlnore, we fix a boundary dcfinillg flmction 0N " X  N+ for OX, i.e.. ON is 
a smooth function such that OX = {0 = 0} and dON # 0 at OX. 
Let ç(X) be the Lie algebra of all edge vectorfields, i.e., ail smooth vector fields 
on X that are tangent fo the fibers of ç at lhe boundary [22]. Then the vector 
fields in 
v(x) := {e v(x). os e o.c(x)} 
are ('allcd bered cusp or simply ,-vector elds. Note that the defilfition of the Lie 
algebra ç(X) depends slightly on lhe choice of the boundary defining function  [24]. To give a local description of -vectol" fields near the boundary, let 
(4) (x. y, z) - X  U  + x v x  
be coordinates of a local product decomposition near the boundary, i.e., x = ONlu, 
y are variables on the base }" lifted through ç, and with z corresponding to coor- 
dinates on the fiber F. Then I"  ç(X) is of the form 
 m 
v(,v.z) =(.v.z).o« + ,(,,z)o, + (,v,z)O, (,. :) e u 
=I k=l 
with coefficient.s a, bj,ck smooth up to x = O. Thus, by the Serre-Swan theoreln 
there exists a smooth vector bmdle eTX  X together with a llmp e  eTX  
TX of vectol" bundles over X such t hat 
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We call "TX the Op-tangent bundle, and its dual "T*X the dp-cotangent bundle. 
Note that the compactness of X is hot necessary for tlle existence of the -taugent 
bundle 4TX. 11 the special case when the fibration ç = id  OX = Y  " is 
trivial, -vector fields are also kuowu as scatterin9 vector fields [28]; accordingly, 
lu that case the -tangeut bundle is called the scatterin9 tangent bvndle, and we 
write 4TX = SCT,¥. By the very dcfinition, the restriction of the canonical map 
t4 " 4TX  TX and its dual t  T*X  4T*X to the iuterior A'0 := X  0X are 
isomorphisms. Thus, the canonical symplectic form w on T*X]xo can be pushed 
forward to a siugular closed 2-forln w4 on 4T*X[xo giving 4T*Xxo the structure 
of a sylnplectic manifold. With respect, to local coordinates (x, y, z) as in (4), and 
the associated local coordinates (x, y, z, , q, Ç) on 4T*X]t, w4 is given bv 
dx dy dx dg 
so w4 6 C(4T*X, A2(4T*(4T*X))). The nssociated -Poisson bracket {f,g} is 
locally given by 
(S) {f, 9} = .rO.fO«g-.r20«gOçf + a'OufO,,g-.rOugO, f 
+o:foça- o:aaçf +  (,o, fo«a- ,o,,ao«f) . 
Let us denote the kernel of the map tlox  TXIox  TA'Iox- by NOX. Note 
that NOX is locally generated by the vector fields 20 and z0u; hence, NOX 
is a vector bundle over OX. Let I'0X  OX be the vertical tangent bundle, i.e., 
the kernel of the differential Tç  TOX  T]'. Then we have the following short 
exact sequence of vector bundles over OX" 
(6) 0  NOX  TX  I'OX  O. 
In fact, the bundle NOX is the pull-back ç*(NY) of a vector bundle NY over 
Y. Since this bundle ploEvs an important role in our computations, let us reeall 
one way of constructing it. For small e > 0, let  := " x [0, e), and let 
be the corresponding scattering tangent btmdle. Then N" :: «T%[-x{0} h 
the desired properties. Iudeed, with respect to the local coordinates (4), the map 
O" NOX  NY with O(z20l(0,u,)) = z20(u,0) and O(xOu(o,u,)) = z0ul(u,0 ) 
is well-defiued, does not depend on the choice of local coordinates, and makes the 
following pull-back diagram commutative. 
 NOX  
OX » Y 

The fibered cusp double space. For the defiuition of the fibered cusp calcuhls we 
need a compact nlanifold with coruers that is obtained by blowiug up a sequence of 
p-sublnanifolds of the product X 2. For ttle concept of blowiug up p-subnlauifolds 
we refer for instance to [10], [28] and the forthconling book [25]. 
Let pî " Xî := [X2; (0X) 2] --, X 2 be the b-blow-up [27]. By [27, Lemlna 4.1] 
the b-front face fro, i.e., the new boundary face of X obtained by the b-blow-up, 
is canonically diffeolnorphic to the product [-1, 1] x OX x OX. Note that we llave 
fixed a boundary defining filnction ON to define the fibered eusp structure. Then 
B := {(O,q,q')E ffb = [-1,1] xOX xOX : ç,(q) = ç(q')} 
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is a p-submanifold of X, and 
g- x := [xî; ] 
is called the fibered cup or briefly thc -double space. It is a compact manifold 
with corners up to codimension two. Let A ç X  be the diagonal. Then the 
.k- 
subnlanifold A := (d)-(A  (OX)) "4 is said to be the lifled diagonal, and 
the boundary face ff of X that meets the diagonal is called the fibed cusp or 
simply -front face. The -front face is of great importance for understanding the 
properties of the fibcred cusp calculus near the boulidary. As explained in [23], 
[42], ff, is canonically diffeomorphic to 
(7) ff.  OX xr O.X" = OX xr OX xr '. 
where eOX (resp. e-) is the radial compactification of 
. ' w" of fie is canonicallv 
as explained iii Appendix A (.onsequently, the interior ,, . 
diffcomorphic t« OX x- NOX = OX x} 0X x} ¢N)'. If necessary, we use local 
coordinat es 
X -- X  -- 
(8) r = '+ x', S (.+.r,), 
nt {r = 0}. The lifted diagonal is then given bv A {S = 0. U 0," = 
:'}. With respect to he deconposition (7) of the -front face ff, the variables 
S Ç Ns and U Ç N thon correspond to the lincar variables in the fibers of N} . 
The -densi*9 bundle. Finally, for a density bmdle that is adapted to the fibered 
cusp calculus, apply the Snlooth functor a of a-densities to the -cotangent bundle 
T*X to obtain the bmldle Q(X) of -a-densi¢ies. The choice of local product 
coordinates  in (4) trivializes eQ(X)[. A non-vanishing section is given by 
On the -double space Xg, we use the -kemel-hag densi¢ budle KD/= := 
O-(2+n)/[ /2( X 2  Il is COnlpletely characterized bv the space of its Ce-sections 
[27. Lelnma 8.6], namely 
Here,  usual, Ou. " X + + stands for a defining flmction of the O-front face 
ff. 

The fibered cusp calculus. The fibered cusp calculus is defined by characterizing 
the singularities of the Schwartz kernel associated to a bounded linear operator 
A : E(X, OQ/2)  C-(X,o/). A convenient description of the Schwartz 
kernel kA  C-(X ,e/ N oQ/) is possible when replacing the double X  
with the O-double space Xg. Recall that the blow-down map   X + X = 
induces via pull-back and duality an isomorphism 
( ). . c-( x, -D/ )  C-( x  , «/  «'/). 
ç call the image h of kn under t, his isonlorphism the lifled Schwaz kernel of A. 
Definition 2.1. The space (X) of fibered cusp or briefly O-operators of order 
m Ç Ç consists of all continuous linear operators 



HOMOLOGY OF FIBEIRED CUSP OPERATORS 3017 

whose lifted Schwartz kernel hA belongs to the space Iï(X, A: IçD/2) of clas- 
sically conormal distributions that vanish to infinite order at ail boundary faces of 
X$ other than the front face ff4, and are extendible across ff. 
A fibered cusp operator A E (X) nmps Eoe(X, /) continuously into 
Ç(X,I/2); thus, composition of operators as well as conjugation with arbi- 
trary complex powers of the boundary defilfing function 0 are well-defined. Since 
we get the lifted Schwartz kernel of 0A0  by lmfltiplying the lifted Schwartz 
kernel of A  (X) by a smooth function on the interior of X that extends 
smoothly to the front face, is identical to 1 at 4 and polynolnially bounded near 
ail boundary faces other than the front face, we bave 0}(X)0  = g(X), and 
we can define for k  C 
 '(x) := (x)= (x)  . 
Composition of operators leads to a bilinear lnap 
m',k' im+m' ,k+k" 
that defines on @ (X) :=  Uz @'k(X) the bi-filtered algebra structure 
already considered in the introduction; in the sequel we use the notation introduced 
therein. We emphasize the negative sign in the definition of the second filtration, 
whose purpose is to make both filtrations increasing. 
'(x) 
The filtration by the order of vanishing at the boundary on the algebra @ 
induces in particular filtrations Oll o and Ao. namely 

with quotients 
[0 mI := 0/0--1 = 
q1,.An,[k ] :: ,k/,k--1 = 
As above, we let m] := @mm m], and so on. Similarlv,. the filtration of 
 '(X) by the operator order induces filtrations of Z and A, and we use the 
corresponding notation without any further comments. The usual symbol map for 
conormal distributions together with the fact that the density bundle of T*X is 
canonically trivialized by the sylnplectic form leads to the principal symbol lnap 
for the fibered cusp calculus, 
m-l'k(x)  'k(x)  s[m](*T*X)  O, 
which is multiplicative in the obvious sense. Let *X be the radial compactifica- 
tion of the -cotangent bundle T*X in the sense of Appendix A. Then the choice 
of a defining function 
for the -cosphere bundle 
s*x := (T*X k {0})/+ ç *x 
yields an identification of the space S[](¢T*X) with the space C(¢S*X). 
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The normal homomorphism. In addition to the symbolic bchavior at the lifted di- 
agonal, the behavior of the lifted Schwartz kernel hA of an operator A  (X) 
at the -front face deternfines for instance the Fredholm properties of a fibered 
cus 1) operator. So, let us consider (A) := A[ff¢ directly. As shown in [23], the 
distributional density e(A) can be understood as a fanfily (A)(9), y  }', of 
pseudodifferential operators of ordcr m on the space ç-1 (y) x N}" that are trans- 
lation invariant with respect to the linear variables in N}', and dcpend smoothly 
on y  Y. The space of ail thesc operators is denoted bv. ss(.-)_¢0-k ". For a 
precise definition we refer to [23]. The family 4(A) is callcd the normal operator 
of A, and e induces a fanfilv of short exact sequences 
(9) o  ON*:I:(X)  *g(X)  *'&«r)_(0X)  0. 
which are multit)lieative in the obvious way. The normal operator extends to A  
.." (X) bv (lefiniug è)(A) :=¢(oA)=N;(Ao) e q(.N,. )_v(OX). 
Fmrier transf()rm al(mg the fii)ers of ¢Nt', i.e., with respect to the variables 
£" and (r in (8), transfi)rms a familv P = p(y, z, D:, Ds, Du) e q(.Ny)_(OX) 
into a family  = fi(v, z, D:, ç. q) (,f ps('udodifferential opcrators on ç-l(y) that 
dep(,nd smoothly on y  ) an(l syml)olically of order m on the dual variables 
 "*} to (S, U) e ¢Nv)" [31]. Let us write qs(.N.y)_9(0N) for the 
corresponding space of Ol,erators, and .,   (A)  s(.N.y)_(0N) for 
the corresponding normal olmrator. For the sake of completeness, let us give a 
description of Mlnilies in sus(eN.y)_(0X) using local data. By a partition of 
unity il, }" we «an assume that the fanfilv b  2s(.N.Y)_(0X) is compactlv. 
supported in an open set I such that ç-(V) = I" x F. Then we have, up to a 
densitv factor in the g-variable, 
where t(F;N x N) deuotes thc space of parmneter-dependent or suspended 
pseudodifferential operators of order M on F in the sense of [40] - sec for instance 
also [9], [19], [21] or [29] for the one-dimensional case. For 1I = -, (10) is equiv- 
Ment to  e C(IT)S(N x N)*-(F). For arbitrary kI e Ç a partition 
of unity in F identifies  (up to opcrators of ordcr -oe) with a finite smn of fam- 
ilies of the form (g, z, D«,, q) for some compactly supported classical symbols 
In the sequel we use these equivalent pictures of the normal operator simultane- 
ously. 

Fiberwise trace for smoothing suspended operators. There exists a canonical trace 
on 02sus(.N.y)_,(0- ), which we review here. An extension of this trace to families 
in çs(.N.y)_,(0X) for arbitrary finite m E Z is constructed in Section 7. As 
explained above, elements of qsu(¢N.y)_,(0- ) are simplv smooth sections of the 
half-density bundle 1/2 (0X x y 0X x :- o-;-Y) that vanish to infinite order at the 
boundary. Thus, inverse Fourier transform along the fibers of oN*Y transforms an 
- -oe (OX) into a smooth section h of 
element h E 

fl/(OX xy OX x)- eNY) = fl/(ff4) 
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,- (0x). 
that vanishes to infinite order at the bomdary, i.e., we have/t ¢ sus(eN t )_ 
Here we have uscd thc identification ff  OX xr OX xy eNt by (7). Recall that 
the blow-down map /3 " X  X  together with the projection X   X onto 
either the left or the right factor identifies A,»  ff, with 0X. On the other hand, 
A  ff corresponds under thc diffeomorphism (7) to 
D := {(q,q,v)  OX Xy OX x)- t" : ç(q) = y and v = 
and we have canonical isomorI)hisms 
 1/2 . . 1/2 . - . 1/2  - 
= ç*t/(l') N tbr(&) N br(0X) N ç tbr( 
() 

 ç.t/2(y) ¢, tue(0X) ® ç*tt/(Y) 
=   (OX) , 

satisf.ving 

Tr(/,1]-/2) : Tr(b2hl) 
for 1 ¢ k]Jsus(¢/V.)')_(O-') and 2 ¢ u(*N-t )_(OX). Indeed. bya partition of 
unity in } we can assume that thc falnily   ç.N.),)_(0X) of pseudodiffer- 
ential operatois is comt)actly supt)orted in an open set 
I" x F. Then we haï'e, up to the density factors,  
and () = fv fa¢ J Tr @(y)(,,/)) «hld dg, where Tr stands for the usua op- 
erator trace on operators on the fiber F. In particular, we sec that Tr Valfishes on 
comnmtators because  has this property. Morver, by gluing the local pieces 
together we obtain the following forin of : 
where ()   (ç t;    t  f.;. (h((, l)) is the canonical trace 
on the space of suspended operators on the fiber -1{9 ). For thc one-suspended 
case the canonical trace Tr was first considered in [29]. 
Moreover, by (12) or directly from the definition of  we see that  extends to 
sus(.N.y)_(OX) as long as BI < -N= -dira X. 
Commuting fibered cusp operators with log 0- Since A 
act as bounded operators on d(X,Q/), we can consider their commutator 
[A, log0N]" doe(X,Q/)  d(X, 
Lemma 2.2. The commutator with log 0 yields a map 

where (11) follows frolll the fict that the nol'nlal direction to the Ioulldal'y in 
e/Y1- = S«TI'[yx{0} leads to a canolfical lrivial factor in thc fiber half-density 
t/2 «, . 
bundlc fibe( N )t. Thus, we can integratc h]ax over 0X = A,,  ffv and 
obtain a nmp 
X 
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Pro@ Multiplication with logc0N is a local operator; hence we can restrict our- 
selves to operators A with lifted Schwartz kernel hA supported in a coordinate 
patch of Xg. Let us first assuine that this patch is close to the intersection of 
the lifted diagonal &4 with the fibered-cusp front face ff4.Ve use local coordinates 
(r, S, U, yz, z ) as in (8). A straightforward coinputation shows that the lifted ker- 
(+s (+s 
riel nB of B := [A, log 
vanishes to first order at A = {S = O,U = O, z = z'} and ff = {r = O} and is 
polynomially bounded for [SI  oe, we obtain B  -'-I(x). The remaining 
cases are silnilar, only simpler. 

The scatt«ring calculus. As a rule, whenever the fibration (p  OX -- t" is the 
id(,ntity lnap, i.e., whelmver the fil)er of (p is a point, we replace the identifier 
with se and talk al)out cotteriTg instead of fibered cusp. 

3. SCATTERINC AND tP-OPERATORS OF ORDER 

\Ve are going to show that fi)r slnall  > 0 there exists a noi»natural injective 
algebra homomorl)hism from smoothing scattering operators on é to the algebra 
of smoothing fibered cusp-op('rators. This morphism (-) will be used in Section 4 to 
compute the Hochschild homology of the smoothing boundary ideal '2"0. 
The choice of a normal fit)ration near the boundary ?)X together with the 
boundarv defining function cON yields an open neighborhood Ue of the boundary 
and a diffeomorphism [,Ç -- OX x [0,¢) for some e > 0 such that cON[U corre- 
sponds to the projection ON x [0,«) --, [0, e) onto the second factor. Moreover. 
(U«),î) := (/3)-(Ue x Ue) C_ Xg is an open neighborhood of the front face fie, and 
for k  C we tan define 

I.l)e ((U¢).)) " : = 0 at \ ff. . 
q,««ç, (g) := cO. - e C((U)... 1/2 2 o(g). 

where cOl%  (U«) -« + is a defining flmction for the W-front face. Because of 
the compact support, condition, ,c¢t(Ue) is closed under composition, and the 
canonical inclusion qs;,«¢t(Ue) '--, Pç(.\') is a morphism of algebras. 
Let } := ) x [0, e). Thon Ue - OX x [0, e) "2- d ) is a locally trivial fiber bundle 
with fiber type F. As above, for k  (12, let. us define the compactly supported, 
smoothing scattering operators on )é by 

_, co_ { c(0;2 - ,/2 - _ o(;)L \ »«} 

where ( e)sc is the scattering double space, fisc the scattering front face of (te)se," 2 
}- 2 -- 
and cOfr " ( «)« -- lI+ a defining fimction for ff«. Bv the very definition, the 
fibration Ue --, Y« induces a fibration ç,î)  (U«),î) --, (Y«) with fiber type F x F. 
Choose a fiber half-density u  C(OX. °1/2  such that f,_ u21-(y) = 1 
 "fiber] (y) 
for ail y  t', and let 
/2 X 1/2 
/2 (2) :=_ ve [] ve  COe(ON × Ô.X, Qfiber(O q. ) [] Qfiber(Oq£)) 
._ Coe(OX )< GX. Çfiber(0X )< ON)). 
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We define 
(13) 
(14) 

._ 
,_, ¢i,k(x). 

Before showing that the map Os is lmlltiplicative, recall that the lifted Schwartz 
kernel rAB of the composition of two fibered Cllsp operators A and B tan be 
COlnputed using a triple version X 3 of the qS-double st)ace X. This space is a 
conlpact malfifokl with corlmrs that cornes equipped with three b-fibrations 
where we have used the following projections: 
71-C : X3  X 2: (ql, q2, q3) -- (q, q3), 
rCF : X3 ---* X2 : (ql,q2, q3)  (q2,q3), 
7fs : X3  X2 :(ql, q2, q3)  (ql, q2)- 

Using pull-back a.nd push-forward mder these b-fibrations, wc obtain, up to a 
density factor, 
(15) ,AB (r).  *  * 
((s) () )) 
= . () (-. . 
The -triple spa«e Xg bas been «onstru«ted in [23], [42] in detail. Sinfilmly, we 
can deal with the composition in the scattcring calculus using the scattcring triplc 
space for the triple space construction wc refer to [28]. 
Lt (G) (- }"  
(«)«) bc thc fibered cust) (resp. scattering) triple space of U« 
(resp. ), and denote bv. ug  (U«)  (U«) (resp. ugc " ('e)«3  ()) the 
corresponding b-fibrations. Moreover, the fibration U«  ) induces a fibration 
" ( )c with fit)er type F 3, and the following diagrmn is COmlnutative: 
(1 

()«)s () ()_) 
Most important for us is the fact that the fanfilv of linear maps 
= " %,«0)  ««t «), k  C, 
is compatible with the composition of operators. 
Theorem 3.1. Let A  «xpt(}«) and B  «.«pt(I) be arbitrary. 
ae 

Then we 
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Proof. It suflïces to show that  is nlultiplicative. We use the triple space con- 
struction and (15) for the kernel of the coniposition of two operators. Using the 
colnnmtativity of (16), we obtain 

4. THE GENERALIZED HOCHSCHILD-KOSTANT-POSENBERG MAP 
AND THE HOMOLOGY OF THE SMOOTHING BOUNDAR IDEAL 

We refer to Appendix B for a review of topological Hochschild homology and for 
a sunnnary of the notation that we are goillg to lise in the sequel. 
The algebra nlap (=) constructed in Section 3 induces a map of complexes with 
the smne naine between the Hochschild complexes 
o: c.(%)  c,(o) 
compatible with the boundary fihration.  will use this map o compare the two 
spectral sequences constructed with respect to this filtration, hl fact, we get an 
isomorphisln al °E°; together with the convergence of the two spectral sequences, 
this ilnplies that  induces ail isolnorphisnl on Hochschild homology. 
The graded algebra szl is lmturally isolnorphic to the colnlnutative algebra 
of Laurent polynonlials iii with coecients Schwartz fimctions on 4N*t'. Thus, 
the Hochschild-Kostant-Rosenberg map HIçR induces an isonorphism from the 
Hochschild homology of ,zl to the space of fornl8 (17). AI the saine rime 
induces a map between the Hochschild complexes of ,zl and 
(17) C(" =])   *  * " A-* 
K 
c(% ) 
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Our plan is to const, ruct a map 
" 
nlakillg the diagraln (17) COlnlnutative and inducing an injoctioll of HH. (z]) into 
As( N  ) N An-*(C[z.x-]). The existence of N with the above properties 
will show that O becomes an isomorphisn ou HH(z]). In llis section we will 
only use K on chains in C(z]). hut in Section 6 it will be crucial to apply K on 
certain chaillS in the fil]l graded lomMa.l'V algebra 
 -o , so we give hel'e the general 
COlSt ruct iolL 
AZ'[Z] Let 
hl order to define K we lnllSt filst construct a covariant derivative on -0  
us explain this notion. Consider the sheaf of rings PoI(4N*} x N) of Imlynonlial 
flnctions on eN*I" x N. The algebra 4.,[Zl is a sheaf of Pol(eN*V x N)-algebras 
over eN*t" x N, where the N factor corresponds to the variable x. Consider the 
sheaf ÇP°I(ON*}" x 
A covariant derivative on  [z] is then defined as a lnap of sheaves 
that is Pol(eN*l" x N)-linear in lhc filst argmnent and a derivation in the second. 
More precisely, 
(8) Ve(AB) = (V-A)B + AVvB, 
(19) VI-A = fVvA. 
Fix a COllllection 
6r of fibcrwise halgdensities on 0X and 
a connection in the fibration ç : OX  Y, that is, a rule for lifting vectors from 
Y to OX. Pull back tllese COlllletiolls through the lllapS ON x t- eN*t"  OX 
and eN*I"  t', respectively. They induite a conimctioll in the fibration OX xy 
o/ 
eN*Y  
4N*Y For a vector V tangent to eN*Y, we denote by V- the covariant derivative 
in the direction of the horizontal lift of I . 

Lemma 4.1. 

into the space 

The covariat derivative V maps the space 

Çl/2  
Pro@ Clear in local coordinates in Y. [] 
The elements of the algebra 4 "{°l are fiberwise half-densities on ffë. There are 
two ways of seeing t  eX xy 0X Xy xN*t" as a fibration over OX xy N*t , 
corresponding to the two projections onto the left factor (nL) (resp. the right factor 
(un))- Thus, if A  M '[°] and s  S(OX xy N*: {), let 
As := (n), (Ans)  S(OX x- eN*Ii /  
fiber/" 
o/  
This defines a faithful action of ,[0] on S(OX xy N*} "abr; hence, for V  
PolN*Y, we can define the action of V- on d,[0] by duality: 
Vv(A)(s) := Vv(A(s)) - A(Vv(s)) e S(0X xr eN*t" o 1/  
 "fiber 1" 
Finally, conlbining with the canonical action of 0/0x on C[x, x-1], we get a covariant 
derivative on d,[0] N C[x, x- ] = d,[Zl, as claimed. Properties (18) and (19) are 
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easy to check. The covariant derivative preserves the idem 2- '[z] . Iii the scattering 
case, the definition of Ç- is independent of choices; so we write il simply as V. 
There exists a formula relating thc algebra map  from Section 3 with ç. 
Proposition 4.2. Let A,,. Bo 6 01  C[x, a "-a] = ,z]. Then 
(20) V-((A,,))O(B0) = ('(A0)B0). 
Proqf A littlc care shouht bc excrcised, because thc "naive" statement Çi O(A) = 
(-)(V{A)) does hot hold. Esscntially, the fiberwise density oe2 used to dcfine (-) is 
hot parall«l, whilc ils vohllne (whi«h was «hoscn to be 1) is.  can clearly reduce 
ourselvcs to proving the statement fl»r A0, 0 C seZ0]. Note that *Z °] acts faithfully 
on S(+N * Y) (by specializing Yom  to scattcring the action described above). On 
thc gra«h,d ideals, the mal» (-) tan be described in terres of these actions by 
{-)(.4o)(.) = *(.40(.(, ,))) -. 
where n is the projection OX x 4N*l  N*I. Now n.(n*(so)v 2) = 0 for all 
.so C S(g'N*}'). I,ecause oe2 has volume 1. Using this. the fact that .(Vt-(v2)) = 
V(.(t2)) = I'(l) = 0, and the id«ntity Vv(*s0) = *(V(so), valid for any 
c« mlwct i« m. thc resl of lhe proof is straightfl>rward. 

VG, Cml now ch'fine the mal» K. This is (lonc as in [35] along the lines of a map 
co, istr,,cted i,i [33]. Le, A = A,, ...  4t- be a k-H,whs«hild chain in C(z]). 
br l  ..... I polynomial ve«tor fields (or derivations in Ç[x, x-]), define 
Here  denotes the fiberwise trace of smoothing suspended operators as explained 
Section 2. 

Remark 4.3. It is clear fronl the definition that K is still well-defined for tensor 
products A -4o ®... ® 4,. with .43 E  '[z](X) such that mi  C and 
>R(m0 +... + ml.) < -m = dira(F). 
because the fiberwise trace still nmkes sense. In that case /x'(A) is a form with 
SVlnbol coefficients of order .(m0 + ... + mk) + m. 

\Ve will lise this fact later on. 

Proposition 4.4. (1) K o b = 0. 
(2) K o -0 = HIçR. 
(3) Iç is inje«tiv« on HH('V2-[0x] ). 
Proof. The first t.wo claires are straightforward, using Proposition 4.2. For the 
third, we lise the strategy of [35, Proposition 5.4.9] to show that everv cycle is 
homologous to a cycle that belongs to the inmge of 6). Then the first two assertions 
together with the Hochschild-Kostant-lRosenberg theorem yield the result. [] 
The first two statements of Proposition 4.4 show that (17) commutes. From (17) 
it follows that O is injective on HH([o]), because HIçR is an isoinorphism on 
homology. The third statement of Proposition 4.4 imp]ies that 6) is also surjective 
front HH(e2"[o]) to HH(¢2-[0]), Note now that these homology groups are just 



HOMOLOGY OF FIBERED CUSP OPERATORS 3025 

°El(So), respectively °EI(a'Zo). In conclusion 0 becolnes an isolnorphism at the 
level of °E1 between the spectral sequences of s0 and 
The homology of Sfo is a pal-ticular case of [14, Theoreln 4.15] in the double-edge 
case. So we have 

Proposition 4.5. 
HHk(ïZo) H2n+2-k(¢bN*y) 
However. thc proof ri'oin [14] is quilç inexplicit, since it lnakes use of a Çech 
complex adapted to Hoçhschild hollology.  Call give a bel.ter proof of this result 
in the scatlering case. 
Forln the spectral sequence °E with respect lo lhe boundary filtration.  have 
just seen that 
0 1 s i+j  Ai+j-lt4nr.-)x-l-id.r. 
E«(%) = A s ( N*'). - 
 claire that, up to sign, 
(21) d  = *s«d*sc, 
* - * is the conjugate of 
where d is the de Rham diflbrelltial, and *sc := (»c) * Zsc 
Brylinski's symplectic dualitv operator by the dual c of the scattering structure 
lnap zsc  «TtW Tt. Indeed, this follows as iii [3] frolll the fornmla of the 
normal operator of a COnllnutator: if A Ç s%,  Ç s,Z" then g+J-1]([A. ]) = 
{[i]A [J]B t wllere { }c := 01c)-1{ }'c is the scattering Poisson 
bracket. 
We now note that lhe isolnol-phisn *sc splits according to x-degree as follows: 
k--1  r* - l--1 
(ee) *«" A(oAr*}')x t A s (  } )x dx 
2n+2--k  * - l+k--n--1 2n+l-k  * - l+k-n-2A 
,A s ( N) oAs ( N) ,,«. 
The homology of holnogeneous forlns iii x is concentrated in holnogeneity 0. To- 
gether with (21), it follows that 
{ +-i  . - -i ) if j=n+l. 
Ei,j(0) = 0 otherwise. 
Thus, °E(o) degenerates at °E2. The convergence of these spectral sequences 
follows froln Corollary B.9. Now a map between filtered complexes that induces 
an isolnorphisln on E oe is actually a quasi-isolnol'phism, pl'ovided the spectral se- 
quences are convergent [24].  SUlmnarize these remarks as follows. 
Theorem 4.6. The map  induces an isomorphism 
(23) Hn(o)  
 HHk(OEo) = H2n+2-k 
where O(Y)  Y is the orientation bundle of Y. Moreover, HHk(%) inherits the 
filtration by the total x-order, and 
HHk(%) = 0 otherwise. 
Pro@ For the last isonlorphism in (23) we bave used the Tholn isonorphisln 
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5. THE HOMOLOGY OF THE SYMBOLIC -ALGEBRAS 
As Bcnameur and Nistor noticed [2], the homology of the algebra of symbols 
of a large class of pseudodifferential algebras can be computed using the spectral 
sequcnce argument of çbdzi«ki [43] and Brylinski and Getzler [3], [4]. In fact. this 
spectral sequence degenerates at E . However. the convergence of this sequence is 
non-trivial because of the boundarv filtration that plvs a role in the definition of 
the Hochschild chain spaces. 
For the algebra ¢z(M), wherc M is a cosed manifold, the convcrgence is almost 
tautological (see, however, Appendix B). It is also fairly obvious for the double- 
edge algebra [14], because of the vanishing of the double-edge Poisson bracket at 
the boundarv. In thc fibered cusp casc, however, convergence is subtle. X% prove 
in Appendix B a general convergence result which appIies to this ce, fornmlizing 
the corresponding result from [35]. 
The Ho«hschihl chain spaces Ce(4A) adroit a filtration given by the total opera- 
tor order of thc factors in the tcnsor product. This filtration is compatible with the 
houndary map: hence it indu«es a spectral sequence cE. The cE  terre is just the 
hmology of lhe graded algcbra, whi«h in turn is isomorphic to a space of exterior 
forms, the isomorphism bcing given by the H«»chs«hild-Kostait-Rosenberg map: 
«E),(«) = HHi+( ''[]'t, ,X))[i ] _ ,.[i]Ai+i(T*X  {0})[x-], 
whcre as usual the suhscript [i] in A[] stands for thc spa('c of homogeneous forms of 
degree i with respect to the natural +-action along the fibers of T*.¥ 
claire that up to sign the boundary nmp d 
and  is the symplectic duality operator on T*X [3]. Indeed, this can be proved 
 in [4], using the form of the Poisson bracket (5). In fact, this claire also follows 
from the corresponding result in [4] for the interior of X. and then by contimfity at 
t he boundary. 
Notice now that , splits açcording to homogeneity: 
2N--k {T* 
 - (*.x-  {0})  ..[_+, _ ._  {0}), 
and that the de Rham cohomology of homogeneous forms is con«entrated in homo- 
£* , c)ntraçtion with the ra(tial vcctor field  in the fibers 
geneity 0. Indcel. on . [pi 
of T*X is a homotopy between pi and 0. It follows that 
(24) 
« 2  HN-i(S*.¥ 
E i, (A) : 0 or herwise. 
This proves in particular the degeneracy of the spectral sequence, because all 
the higher differentials d  for k > 1 nmst vanish. ç must prove convergence: that 
is, we must show that the cE w terres are isomorphic to the graded groups soci- 
ated to HH(a) with respect to the symbol filtration. This is unfortunatelv hot 
obvious: the E ° terre of the spectral sequence has infinitely many nonzero com- 
ponents along the diagonals {(i,j) G Z2; i + j : constater}. So standard diagram 
chasing will produce infinite sums of Hochschild «hains. Such sums make sense 
(i.e., are ymptotically summable) only if they are uniformly bounded in the three 
filtrations from Appendix B. Moreover, as for anv spectral sequence, we only get 
information about filtration quotients of the homology; we in«lude in the definition 
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of convergence the condition that the "residua[' part of the homology (i.e., those 
classes that adnfit representatives of arbitrarily low filtration order) vanishes. 
 are going to propre the convergence of «E(a«) using Theorem B.4, by showing 
that assmnptions H1 H6 hold. Fil'st, the fibered cusp sylnbol algebl'a satisfies the 
hypothesis of Lelmna B.6: so H4 and H5 do h»ld. H6 is obvious. Condition H2 
tbllows ilmnediately fl'OlU (24). likewise the first part of condition H I (the existence 
i+,z;z such that b() + 
of 6 t. 
For H3. we study the spectral sequence aE (relative to the symbol filtration) of 
the graded algebra 4,[z] of 4« with respect to the houudary filtration. Recall 
froln (9) that oMS'[z] 
Proposition 5.1. The spectral s«qten«e «E(42 [z]) of the Hochschild complez of 
4,[z] degenerates al aE2. 
Pro@ This result is essentially contained in [33]. Cousider the short exact sequence 
of algebras 
whel'e B» is the ideal of those sylnbols in ,[Z] that valiish rapidly at the vertical 
sul»-bulidle of *T'X, and (25) serres as the defiliition of the algeina Fs. This 
short exact sequence induces long exact sequences of cE  terres, but these actuallv 
decompose  short exact sequences of honiogPlieOtlS differential fOl'lllS 
o  E(B.)  » ..« ,  E(F.)  O. 
Thus we get long exact sequences both on the CE  terlns (froln the short exact 
sequences of cE a terres) and on Hochschild holnology (induced froln (25) by H- 
unitalitv of B). 
There exist many exterior derivations on the algebra 4A'[z], namelv, the covari- 
ant derivatives çv of Section 4. for I" either equal to 0« or a vector field with 
polynonlial coefficients on eNI These derivations act on tlle Hochschild colnplex 
by contraction and bv Lie derivative (sec [20]). These actions descend to Hochschild 
homology a.nd to the spectral sequences, and thev colnnmte with the differentials. 
Since the polynolnial functions f on eNI and in the variable x are central iu 
4A'[z], we tan define au «exterior product" action of such a fimction f on the 
Hochschild complex bv 
k 
a0 @... @ a  (-1)a0 @--- @ ai @ f @'-" @ ak. 
Again, this operation commutes with the Hochschild boundary map, and hence 
with all differentials in the spectral sequence. It is straightforward to check that. 
for a polynolnial functiou f as above, 
[evv. f A = V(f)I. [Lvv,fA=I'(f)A f A9A= -9 Af A, 
where V(f) is the V-derivative of the fimction f in the direction of the vector V. 
Also, on Hochschild homology, evvevv = -evvevv. This was shown by hand in 
[35], but it follows directly from the product structure of Hochschild cohomology 
HH*(A, A) and its action on homology (we are grat.eful to Colin Ingalls for this 
argmuent). 
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These algebraic properties of ¢4 '[z] and the explicit computation of «E2(Fsus) 
and «E2(Bus) similar to [33] are enough to show that (i) the spectral sequences 
«E(Fsu) and E(Bus) degenerate at cE 2, and (ii) the boundary map in the sequence 
of cE 2 terres vanishes. Inductively, we get a long exact sequence of cE  terres that 
actually splits into short exact sequences, for p 2 1. The Five Lemma shows that 
«E(A '[zl) also degenerates at 
By Corollary B.8 we see that H3 is fulfilled. Now we use the notation of Appen- 
dix B to prove H 1. 
i,j;l( 
Proposition 5.2. Let a 
i,-i, i.e. such 
that d°[a]«Eo = 0 andd[a]«E  = O. ffk-i ¢ N andl  l+max{i+ï+,j+)+},° 
p[i] ,j + 1 ;l 
«za(t as a cbain in . . 
Pro@ Let [a]«E ( 
is closcd, since a is a cy«h,, ami it is exact because k - i ¢ N. Moreover, 
('- 
where « is the canonical 1-form on T*X. Bv Lemina B.7 the forin 
can be representcd bv a «hain 
Thus ttl also hohts for *M; so we cm apply Theorem B.4. An entirely similar 
analvsis is done for the algebras *Ao,« and eXC. k" sunmmrize the results in the 
next theorenl. The subscripts rel (respectively, abs) denote de Rham cohomolo" 
of forms vanishing to the boundary (respectively, smooth up to the boundary). 

Theorenl 5.3. 
HHk(4«) 
HH k (¢Ao,,, ) 

Moreover, for i ¢ k - N, HH(,4«)[il and HHk(¢.Ao.,,)[il vamsh, and HHk(.A«) 
( respectvely HH(.Ao.,, ) ) can be represented by chains in c-N'-'-(Ao) ( resp. 
 C-,-"-' (Ao,«)). 
Proof. The isomorphisms follow from Theorem B.4. Bv analyzing the @-symplectic 
duality operator ,e (the homogeneities given in (22) m'e valid for absolute forms 
with symbol coecients as well) and by Lenmm B.7, we get the indices with respect 
to the boundary filtration as claimed.  
}ç remark that the long exact sequence in Hochschild homology (which exists 
since CZ is H-unital) coincides with the long exact sequence of cE 2 terres induced 
from the short exact sequence of cE 1 terres, which is just the de hanl absolut 
relative cohomology long exact sequence. 

6. THE BOUNDARY SEQUENCE 
In order to compute HH(¢Ao), consider the short exact sequence of algebras 
(6) 0- o- CAo- Ao,«-, o. 
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We have already computed the homologies of the two extremal algebras. Bv H- 
unitality of 0 (a.n issue that we are hot going to stress here), there exists a long 
exact sequeiwe iii Hi)«hschihl homology in(luce(l fiom (26). This long exact sequence 
is completely determined by the i)omularv maI)s 
(27) " HHk(*o.) HH_,(%). 
When looking al the associated graded algebras with respect to the bomdarv fil- 
tration, the result is quite simple, in the sense that all bomdary maps vanish. We 
have alreadv mcntioned this fact in Proposition 5.1. This says basically (as shown 
in [33]) tha.t there is no mmlog of thc index map for suspcndcd opcrators. It follows 
that the bomdary maps  decrease the bom(lary filtration ordcr by at let 1. The 
only relevant dimensions are for k  n + 2; othcrwise HH-I(O) = 0. Nov note 
that HH+(¢Ao,) has representatives in bomdary filtration k- n (Theorem 5.3). 
Let  be a homology cls in HH+I(¢Ao,). By H«mitality, there exists 4 £ 
C+I(¢Ao.) representing the class o such that b(A) £ C(o). Moreover, wc can 
assume that 4 £ +1 t «,. From thc fact that the bomdary map in the 
k-n- 1 
associated graded algebras vanishes, we can also assume that b(A) £  (c)- 
The boundary mal)  is dcfincd simply by () := [b(A)].  do no know in 
general how the isomorphism from Thcorem 4.6 a.ssociatcs a cohomology class to 
a cycle: however, we bave bccn able to show that the bomdarv filtration ordcr of 
the cycle b(A) is k - 11 - 1, which is exactly the filtration order wherc HH(o) 
is concentrated (Theorem 4.6). This means that [b(A)] can bc expliçitly computed 
from its top component: 
2n+2-k  
lb(A)] = ,«ç(b(A)_._) e  (   C). 
Now use the Thom isomorphism between Schwartz-coecient de Hham cohomology 
and cohomology of the base with coecients in the orientation bundle: 
[b(A)] = N/Y ,»«lç(b(A)[_._])  H'+-(t"  £. 
Now let Q be a positive -operator of order (1,0) and let Q be its complex pow- 
ets. The construction of complex powers in the -setting is briefly discussed in 
Threm 7.1. Notice the identity 
Q:b(A) = b(QA) + zQ:ogo(A) + O(z), 
where Oogo(A) is the action of the exterior derivation A  [logQ, Ak] on the 
Hochschild complex given explicitly by (see also [20]) 
A0 @... @ A  [logQ, A]A0 ...  A_t. 

We can then write 
(28) [b(A)] 

NY/Y 
f lV,./y *sclç(b(Q%4 )tk-,-11)lz=O] ,ol 

h-Res_-=0 



3030 ROBERT LAUTEF/ AND SERGIU MOROIANU 

where the sui>script [0] denotes the part of homogeneity 0 in .r. We first examine 
the second terre in (28). Bv Fubini's formula, it equals 

(2r),. Resz=0 r.x/v 

We claire that this is just 

[o] 

(2re) TM s. x/v 
lndeed. ,4)tth'R(Q:ClogQ(A)) is a fil'm with symbol coetficielitS on ¢T*X of homo- 
geneity z. So the residue only depends on thc principal svmbol and is given by the 
aboç fhrmula by a standard computation in Xk»dzicki-type forlnulas. 
We claire now that the first terre iii (28) vanishes in cohomology, i.e., it is exact. 
This woukl be clear if b(Q: 4)[-nl were equal to 11, since iu that case the saine 
colnpltatioll that gives d  in OE(O) would give 
(29) *«Iç(b(Q:A)[__q) = d * Iç(Q:A)[_]. 
This lmMs ft,r values of z of small enough rem part. By the Hodge theoreln, the 
space of exact forms is closed in the C  topoloKv; hence the regularized value at 
z = 0 is also exact. In gencral, b(Q  4)[_]  0 modulo an entire Nmily of chains 
of order (N - k - 2 + z, [k - ni) that vanishes at z = I): hen«e the identity (29) is 
shown to be valid at z = 0 s in [35]. 
Proposition 6.1. In terres of the identiflcations of Theorems 5.3 and 4.6. the 
boundar9 map (27) is given by 
6.: 1 
(2)  S.XlV 
The two copies of S 1 in Theorem 5.3 correspond to a concentrated way of writing 
the de Rhmn cohomology of homogeneous forms in the vm'iab]es x and r. By 
dr 
Proposition 6.1 the bomdary map sends H2N-k-t(S*A'ox x çl) to 0: hence 
(fronl the long exact sequence induced by (26)) this space lies in the image of the 
canonica] map HHk(4o)  HHk(4o.,). The dimension of the sphere fibers of 
the fibration çS*-¥1ox  OX equals the dilnension of the base. and so from the 
Leray spectral sequence it follows that *'*Xl0x is cOhOlno]ogicany a product: 
For k < n.+ 2 < N we deduce that 
H-(çS*A'Iox x S 1)  H-+(OX x S . O(X)). 
&Ve get from here a characterization of HH(o). Let I.v_k+t be the push-forward 
map (integral along the fiber) 
I-+1 : Hn-+(0A " x S,O(.¥))  H-+(Y x S,O(t')). 
Theorem 6.2. 
HH(çAo)  H--(S*Xlo - x S )  ker(I_+)  coker(I_). 
Pro@ In the long exact sequence i,,a,,«a by (26) we write HHk(eo) as ker(Sk) 
coker(Sk+t). These spaces were identified above. 
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Corollary 6.3. 
to a constant) continuous trace on 
Pro@ From Theorem 6.2 for k = 0 we get 
[N+I 
coker(IN) 

The dimetsion of HHo(¢.Ao) is 1, and  there exists a utique (up 

= 

Only lhe last equality lleeds solne exI)lanati()ll. The illtegral mat) 
Iox : lIN(Ox x S:, O(X)) -- Ç 
is m isomorphism. The saine is truc for Iy : H'+(Y x S ,0('))  C. But 
Iox = Iy o IN; so IN is also an isomorphism.  
 renmrk that the boundary ma l) and the homology of ¢0 (let)end strongly 
on the geometry of the boundary fil)ration. 

7. TRACES AND GENERALIZED RES1DUES OF TliE FIBERED CUSP CALCULUS 
The computations of the prcvious sections show that dimc HH0(A) = 1 for 
A = cZ, ¢Lro, ¢.A«, (:A0, ¢.Ao,«; thus, there exists, up to a multiplicative constant, a 
unique trace on these algebras. We are going to identifv these traces as residues 
of the analvtic continuation of a "doul)le-zeta'" flmction, and we are going to give, 
additionally, explicit formulera. The corresponding results about traces for the cust) 
calculus were obtained in [34. Section 5]. Some of the proofs depend on fornml 
nmnipulations with zeta fimctions that can be round in [34]; so we will only state 
those results and leave the necessary changes of the proofs to the reader. 
Theorem 7.1. Let Q  ko°(.\ -) be a positive (-operator, possibly with bu, ndle 
coefJïcients. Then the family of complex powers Ç  z - Q= is a holomorphic 
family of d2-operators, Q:  kg'°(x). 
Pro@ The complex powers of Q are constructed using the method of Bucicovschi 
[5], who extended the proof of Guillemin [12. Theorem 5.5] to the case of non- 
comnmtative symbols. We assume the reader to be familiar with these two papers 
for the purpose of this proof. 
The algebra q(X) has two symbol maps, which nmst be treated simultane- 
ously. We first note that the complex powers of A/'e(Q) form a holomorphic fam- 
ily of suspended operators, AIe(Q) z  q2u(.vy)_,(OX). This follows modulo 
-u*Ng" (OX) ri'oto [5, Proposition 1.4], and is corrected to a true familv of 
( )-- 
complex powers as in [12, Section 5]. It is clear that the principal (conormal) 
symbol of Q admits complex powers, and that 
«(J%(Q):) 
This allows us to start Bucicovschi's induction argmnent: there exists a holonorphic 
fmnily Qo(z) with Q0(0) = I. Q0(1) = Q, «(Qo(z)) = «(Q): and Af¢(Qo(z)) = 
Af(Q) z. This implies that 
Qo(z)Qo(r)Qo(z + r) -t = l + R(z, 
where R(z, r)  k0'-(X) is holonorphic in the two variables. Notice that in 
[5] the error symbols R i are sections in a von Neummm algebra bundle, whereas 



3032 ROBERT LAUTER AND SERGIU MOROIANU 

here they have a componelt in the Fréchet algebra us(,N.)_(O.\') (the bound- 
arv symbol). Nevertheless. there is basically no modification needed to prove [, 
Propositions 1.3, 1.4] lu oto" case: a new argunwnt is only necessary to show that 
the mai» 
k- ' " (it reduces then 
is surjective. This surjectivity is t.rue modulo ss(y)_(0X) 
to Bu«icovschi's case of nmtrix vahwd symbols), and also on ss( )_(ON). 
Aswnpt.otic completeness of s(..)_(0X) modulo the ideal (N )_(OX) 
ends the proof of the induction step. 
By induction and symptotic completeness of (X) modulo oe'-(X), we 
get an approximate complex powers fa.mily Q(z) modulo  (X), i.e., 
ç(z)Q(ç)Q(z + ç)- = 1 + R(z,ç)  I + '-(X). 
Again s in [12, Section 5] we show that Q(z) differs from the true complex powers 
fiunily Q: by a holom«,rp]fic family iu ',» (_ç); so il follows thal Q: 6 °(x) 
is a holomort)hic family. 
In particular, we see t hat lhe complex power Q: is again a -operator of complex 
order z 6 C. Extending the approacl of Guill«min [12] and Bucicovschi [5], complex 
powers could recently le constru«ted sinmltaneously fir a large class of boundary 
fibration structures and nou-compact manifolds [1]; however, it is necessary to 
stress that in this generality complex powers are realized in the so-called Guillemin 
complet ion of the calculus, which contains slightly more smoot hing operators than 
the original (small) calculus. 
Proposition 7.2. Let  ç C  be open and connected with [R, 
for some R 6 , and let A " t  °'°(X) be holomorph,c. Then the functions 
wzth T  (,z)  Tr(A(,z)oç -) and T  (,z)  '(A(,z)ç-O), are 
holomorphic. Moreover, here ezist memorphic functions     Cwih at 
most simple poles af  = mo + N - (. ( G No. and z 
which coincide with T on the component of  containin9 [R',oe)  [R', ) for 
some R  G . 
For A G °'°(N), we let Zou,Q(A ) be the meromorphic extension of the 
holomorphic function 
{ez > #0 + n + }  {Re > ,n0 + N}  (,z)  (Ab-), 
which exists by Proposition 7.2: then (. z)  zZo,Q(A)(,z) is regular in 
neighborhood of 0 6 C z, and we can define 'o.,(A), Tro(A) and Tr,(A) bv 
+(, ) + :w'(, ) + w"(,), 
where IIçII". W" are holomorphic near 0 G Cz (and hot uniqueS). Because of 
(Aoç - - Aç - 0) = Tr (A(id -ç- 0ç 0) 0 Q- ), 
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Furthermore, let  be the ca.noical singula.r symplectic form on oT*X, and let 
us dcnote by i (resp. 'io) the contraction wit.h the radial vector field  in the 
fibers of T*X (resp. with the canonical ve«tor fiehl 0, normal to the bomdary). 
Recall that p« - 4*X  + denotes a defining function for the -cosphere bmdle 
S*N. Moreover, the pull-back of t.he boundary defining fimction ON X 
under the projection *X  + yields a defining flm«tion for the face 
again denoted by ONfor simplicity. 
In the test of this section, we smnmarize the properties of the flmctionals 
o, and 
Proposition 7.3. The double-residae Tro,. " 'z(x)  C is a trace functzonal 
that is independent of the choice of Q. Moreover, 
To,«('Z(X) + '-(X)) = o. 
Thus. Tro,« defines a trace functioal on o.«. If 4  o,« is represented by 
kSko,msmo Am,k with Am,k  eN e«  ç o A IOX ). then 
(2w)-N [  - N 
Tro.«(A) 
J4 
S*X[OX 
The "unique" traces on 4 (resp. o) are then given by the composition of 
ô,« with the nat, ural projections   o,« (resp. oô  
Since ZQ,(A) is entire for A G Oe'-Oe(X), the linear flmctionals « and 
descend to linear functionals on  t-o . 
Proposition 7.4. The restriction « " Z,  Cof « to Z, is a trace func- 
tionai; it is independent of the choice of Q. and eztends the Wodzicki residue for 
the double 2X of X b continuity from operators whose kernels are supported in 
the interior. If A  Z, bas the asymptotic expansion o A, wzth A 
çS[](T*X), i.e., a := (çA)]s. x  pC(4S*X), then 
To understand the functional «, we need to extend the integral 
J 
S*X 
to functions f  pC(S*X) with finite k  Z. For such an f  
let 
J4 
S*X 
Then H0N (f) is holomorphic and admits a meromorphic extension H0N (f) to the 
complex plane with at most simple poles at z = n + 1 - k-j, j G 0- Let 
4-fN fi be t, he regularized value of HoN(f ) at z = 0. Note that 
34S*X 3S*X 
extends (31) as desired to pC(oS*X), but depends mildly on the choice of the 
defining function 
Proposition 7.5. Let A  °'°(X) be arbitrary, and suppose that the residue 
class A + 'Z(X) corresponds to  A with A,,  p'S[](T*X), i.e.. 
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(2çO N J*s.x Pro@ Consider AR for ez > k0 +  + 1. Then the fimction 
Go  {z Ç C" Rez > k0 + ,t + 1}  C" z  Tr«(A0) 
is hololnorphic, admits a meromorphic extension Gow to the complex plane with at 
most simple poles at z = k0 + n + 1 - j, j  N0, and ri-oto (30) we see that Tre(A) 
coicides with the regularized value of Go at, z = 0. Exactly as in Proposition 7.4 
we obtain Go (z) = (2) -N Js.x b(z)-Niç, where the residue class of A0  
qmo.ko-Re 
 (X) modulo 'Z(X) is represented bv the sure 5-o B(z). with 
B(z),  ONR-k,-q[M (çT*X) and b(z) = (Re'" B(z))[.s.x. Since on the other 
hand we bave b(z) = 0a,, a fm'ther look at the definition of the -integral 
ç-fo comph,tes the proof.  

Finally, fir the fimctiolml Tro we lleed fo consider ail extension of the canonical 
trace Tr on tho stm.ce q»sus(«,N.V)_(OX ) to sus(N. )_(0X) for arbitrary finite 
3I G Z (rccall the canonical trace Tr that was introduced in Section 2). To this end. 
fix a positive operat«,r Qox  (*N' )_(0X). and construct the holomorphic 
family C  A  Qôx  (*N-v)-(0X) of complex powers. Note tha.t we 
could take Qox = ,Ue(Q) with Qôx = -Ue(Q) Açe(Q ) for Q e e (X),  
-, -( 
above. Then. for  e s(.N.y)_(0X), the function Fo  ()" A  Tr Q 
is holomorphic in the domain {A G C  eA > I + N}, and admits a meromorphic 
extension to the whole plane with at most simple poles at A = I + N - j, j  N0. 
Let us denote by Oox () the regularized value of Foo x () at A = 0. Bv the very 
definition, Troo x is an extension of Tr that, however, depends on the choice of the 
family Qox. 
Proposition 7.6. The restction Tro :o C of o to o is a trace functional; 
it does hot depend on the choice of Q. If A  o coesponds to o @ 
with .  (.N.V)_(0,Y ). then 
Tro(A) = (Â_(+I). 
For fibered cusp operators A of arbitrary order that vanish sufficiently ft at 
the front Nce, we also have a nice fornnlla for o(A). 
Proposition 7.7. Let A  °'-(+I)(x) be arbitrar 9. and suppose that the class 
A + '-(X) is represented @ 5_(+) @. A 
Then, 
where Qox = , ( Q), 
Pro@ Consider AQ - for A Ç C with e A > m0 + N. Then the function 
HQ  {A  C  e A > m0 + N}  C  A  Tro(AQ -) 
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is hoh)nlorphic, admits a meronmrphic extension HQ to the comI)lex plane with at 
A 
most simple poles at A = Oto +N-j, j  N0, and rioto (30) we see that Tr0(A) co- 
incides with the regularized value of HO at A = 0. Exactly  in Proposition 7.6 we 
see that we have o(AQ -) = ((A)_(+a)) where the residue class of AQ -  
lnodll{O  (X) corrçspOll(Is to k_(+l) Çk with 
=  ,..,,. 
B(A)  .(.r)_(OX). Now, B(A)_(+) = 
A-(n+t)Qox, and a fltrther look a.t the definition of Troox(_(n+l)) completes 
t, he proof. 

Let us point otlt that the expression for '0(A) fol" .4 E P0"k(X) with arbitrary 
/," E Z is more COlnplicated. 

8. COMPUTATION OF TtlE INDEX 
Recall froln [23] that a fil)ered cus I) ol)era.tor .4 G P,'k(.\') is Fredholln between 
appropriate weighted Sobolev spaçes if and onlv if its q-princii»al svmbol oa (v') (A) 
and its normal svmbol A/')(A) are both invertible. Let A be such an operator. 
Then A is invertible modulo  1, I(X), and by a formal Nemnmm series argulnent 
there exists B  qêv'-(X) such that P := id -BA and P'2 := id -AB both belong 
to P-(X), and, thus, are of trace class. With a little bit more effort and using 
a formula for the relative inverse of an operator with closed range which goes ba«k 
at least as far as [41] (see also la] and [11, Belnel'klmg 5.7]) we tan actually find 
B  qv'-(X) such that P and P2 are the orthogonal projections onto the kernel 
and cokernel of A. 
Note that A ® B then defines a Hochschild cycle in HH('z/k'-). This 
simple but crucial observation was ruade bv Melrose and Nistor [34], in the context 
of scalar cusp pseudodifferential operators. An almlogous statement holds for any 
other algebra where elliptic operators adroit pseudo-inverses, lIelrose and Nistor 
extend the definition of this 1-cycle to operators acting on sections of a bundle 
'. By Morita invarimme, the homology of such an algebra is isomorphic to the 
homology of the scalar algebra. Then the boundary map of the long exact sequence 
of the pair (po¢,-, 
P ) applied to this cycle is a 0-Hochschild class of trace 
equal to the index of A. 
A slight extension of this construction is carried out in [35] for a flflly elliptic 
operator between sections of arbitrarv vector bundles g, )r. Replace the cycle .4 
with Tr(ilAP.2 ® i2BP1), where P1, P2 are projections from a trivial bundle C q 
to g,', i,i2 are embeddings in C q, and Tr is the generalized trace fulmtional 
that induces Morita invariance, Tr((aO)® (ho)) =  a 0 ® bji. Of course, Morita 
invariance does hot apply in this general case, since operators frolll  t,o .) do not 
form ail algebra. 
Now we formally follow the computation from [34] modified in [16] and applied 
in a similar situation in [14], and get the following result. The proof is included for 
the benefit of the reader. 

Proposition 8.1. The bouzzdary map  in the long exact sequezzce in Hochschild 
homology induced by te short exact seq,uence 
o ;'- '  -- 
  " / " >0 
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applied to a cycle c E CI(/ /1[/ ) is given by 
C ) Tr(3(c))= Tro(flogeN«)- Tr«(elogQC). 
Here flogeu and elogeu are actions of thc derivation [-,logpg] on Hochschild 
1-chains, namely 
fogeu(A0 @ At) = Ao[A, log 
çloge(A0 @ A) = [Ao,logoN]AI. 
Pvof. Let C be a lift of c to C('z). Then, froln lhe definition, we have 3(c) = 
[b(C)]HHo(ç;,-), and hence 
Tr(b(C)) = (b(C)e.Q-)z=O.=o . 
To av«fid introducing heavy algebraic notati¢m, write formallv C = U  1 . Then 
6(c) = Tr((UV- VU)gQ-)z=o,=o 
= Tr [(t(V- eVez)eç -) + (ç'ç- - V)Ueç-)]z=o.=o 
= " [zU([" ogeu + »F(z))eç - 
+([,,g ç. ri + ())Ueç-]=0.=0 , 
where F (z), F2 (A) me ent ire fimfilies of operat ors of fixed ortier. Bv Proposit ion 7.2, 
the result follows. 
There exist lnore or less canonical choices for B and Q. More precisely, choose 
B to ho an inverse of A up to '- aud Q = (x 4*A.r) /2p modulo '-. 
With these definitions, AS(A) := -Tre ([A, log Q]B) and Tro(A[B, log ON]) are in- 
depelMent of choices. Proposition 8.1 implies the following fornmla for the index 
of A: 

Theorem 8.2. 
index(4) = AS(A)+ Tro(A[B. logON]). 
The local terln can be identified with the coefficient of t ° in the asymptotic 
expansion of the heat kernel of A'A, AA* (this is a universal local expression in the 
full Svlnbol of A, and so we do hot need fo construct the heat kernel of -operators 
in order to define it).  are able to determine the non-local terln Tro(A[B, log ON]) 
explicitly in geometric terres for certain differential operators on manifolds whose 
boundaries fiber over the circle S . 
Fix a connection in the bonndary fibration, i.e., a rule for lifling horizontal »ctor 
fields. Fix a product O-metric on X, i.e., a smooth metric on the interior of X that 
with respect to local product coordinates close to the boundary looks like 
dx 2 d« 2 
(32) x +  + 9, 
where « is the variable in the circle and 9 is a fanfilv of lnetrics on the fibers 
of ¢, independent of x. Let E   OX be vector bundles with fixed Hernfitian 
lnetrics and compatible connections V. Let ,   X be vector bundles such that 
Elox  lox  E+@ E-. Let A  Diff(X; E) be an elliptic ldifferential operator 
of order 1 which near the boundary (for x < e) h the form 
(33) A = «() oh + D i.ço " 



HOMOLOGY OF FIBERED CUSP OPERATORS 3037 

where Dis a fmnily of invertible operators Oll the fibers of the boundary, 7o, = 
1 
V, +  Tr(Lo, g), and 7(.r) is a linear isomorp|fism of E + @ E- that depends on ff?. 
This h.vtmthesis is motivated I)y the fact that when the manifold X and the fibers 
of the boundarv are spin lnanifolds so that thc st)in structures on X and on the 
fibers are compatible, a twisted (P-Dil'aC operator on X with respect to the nletric 
(32) has this form. 

Theorem 8.3. Under the above asswmptiols we have 

(34) 

index(A) 

liln 11(5Æ) 

where the second tewt is the adiabatic limit of the eta invariant of the operator 

[ -i.,'Vo« D* 
D i.rVo, ] 

on OX as x  O. 

For twisted I)irac ot)erators we id(.ntify tllc h)cal terre with the characteristic 
form A(X)ch(T), whose integral ()n X is convergent. A related formula has been 
obtained 1)y Nye and Singer [38] for the Dirac operator on X = 5' x IR 3, where the 
boundary fibration is the projection S 1 x .ç2 --, S 2. The result mmomced above 
requires a sophisticated proof that will be given in [13]. 

APPENDIX A. SYMBOLIC AND SCHWARTZ SECTIONS OF VECTOR BUNDLES 

For the convenience of the readcr, we recall bricfly the definitions of syinbolic 
sections and Schwartz sections of certain vector bundles. Ail important notion for 
that is the radial Conlpactification of a vector bundle [28]. 
It is easy to check that the radial compactification map 

t:{C']N + s+N := {(w,0,w, )  SN.w0 _> 0} "z (z)-l(1, z) 

with (z) := x/1 + Iz[ 2 identifies IR N with the interior of the upper half sphere S+ N, 
a compact mmfifold with bomdary. The following basic observation is essential for 
the definitions of this section. 

Proposition A.1. Let O S+ -- + be a boundary deflning function. The radial 
compactiflcation map RC indttces via ptdl-pack isomorphisms C(SN+)  S(IR N) 
and O-'C(SN+ ) -- sr(]R N) where S(]R N) is the space of Schwartz functios on 
IR v. ad Sm(IR ) the space of classical symbols of order m  C. 

Now let I" be ail N-dilnensional vector space, al,d V := V (9 (V \ {0})/1+. l, Ve 
want to introduce a differentiable structure on V. Choose a linear isoinorphism 
T  V --, IR N, and consider 

{ RC(Tv), 
T- V --- S+ N "ç  lillltoe RC(Ttv), 

ç = ['l  (\ {0})/+. 

Then T is a well-defined bijection that gives I" the structure of a differentiable 
lnalfifold with boundary, diffeomorphic to S+ N. Since each A  GL(IR N) induces a 
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the differentiable stlllCtllr(" O11 1-7 does llOt depcnd on the choice of the map T. 
Everything obviouslv varies smoothly with pal'ameters; thus, the saine COllstr/lc- 
tion carlies over to smooth vector imndles. Let X be a compact manifold with 
corners, and r  I"  X a smooth vector lmndle of rank N. Then there is a unique 
differentiable structure o ç := 1"  (V k {0})/N+ such that " ç  X becomes 
a smooth fiber bulMle: for each q  X, the fiber W-(q) is obtained by radially 
«ompactifying the fibcr n-(q). The bmMle  is called the radial co,pactification 
4 I', wherea.s thc boundary COl,lpOllOllt S(V) :: (I" k {[)})/N+ of 1 w is said to be 
the sphere bundle of I . 
Now let E  X be another vcctor hundlc over X. Note that therc are canonical 
emh(,d(lings RÇ " 1"  I" and RÇ* " n*E  *E such that the following diagram 
çOnllnllt os: 
RC* 
pr pr 
RC 

Definition A.2. Let toc " Ï = -- + be a defining flmçtion for the boundary face 
(V) of 1". A section s  I -- rr*E is said to be a Schwartz (resp. classical) symbol 
of order m E Cprovided it is the pull-i)ack under RC of a section g - 17  *E 
in Ç),,e og"C(,*E) (resp. in Og'"C(Ï-7,*E)). We write S(I. rr*E) (resp. 
S"(I';Tr*I )) for the st)ace of ail Schwartz fUlmtions (resI). symbols) of order m. 
Moreover, in the paper we use the space Sz(l;rr*E) := Ure S"(l/, r*E) of ail 
symbols of integral order. 

APPENDIX B. HOMOLO(;Y OF TOPOLOGICAL FILTERED ALGEBRAS 

Let A be an algei)ra. The Hochschild comt)lex of A (with coefficients in A) is 
defined by Ca(A) := ,4 (k+) with differential d : C'(A) -- Ck_(A) given bv 
b(ao ® . . . ® a¢) := aoa ® . . . ® a¢ - ao ® aa2 ® . . . ® a. + . . . 
+(--1)k-a0 ®... ®a¢_ak + (--1)kakao ®... ®aa_l. 
If A is a topological algebra, it is more meaningfifl to involve the topology in the 
definition (sec [6]). Thus, assunfing that A is a nuclear Préchet algebra, one replaces 
the algebraic tensor product with the projective (completed) tensor product. The 
most useful example is when A is the algebra of smooth fimctions on a compact 
manifold AI. Then the classical Hochschild-Kostmt-Rosenberg map 
HIçR : Ck(C(M)) -- Aa(M), 
o®...®a,-  aod(a)A...Ad(ak), 
has the propert.y Hh'R o b -= 0 and becomes an isomorphism on the continuous 
Hochschild homology. This result extends easily to the case where ,4 is the algebra 
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of smooth flmctions on a manifold with botmdary or with corners, with Laurent 
behavior at some boundary hypersurfaces. In that cse the continuous Hochschild 
homology is isomori)hic via the Hh mat) to the space of exterior forms on the 
nanifohl, with Lam'ent beha.vior at the saine hypersurfaces. The map HIç also 
computes the homology of the fiehl of Lam'ent series in one variable, the ring of 
Laurent series in several variables and of polynomial rings; we implicitly assume 
these fa.cts in the body of the paper. 
Since the product in the topologi«al algebra 'Z(X) is only separately contin- 
uous, we cmmot directly apply the definition of contimloUS Hochs«hiM homology 
as it is defined for instance in [6]. lnstead, we use in a particular case the concept 
of Hochschild homology for topological filtered algebras formally introduced by 
Benmneur and Nistor [2] to cover exactlv the case of pseudodifferential operators. 
The definition (35) of the chain spaces ca.n be traced back to [4]. 
Let A be a bi-filtered Mgebra with topology with two increasing filtrations 
{A'}dez a.dapted to the product structure. The mod,'l for A is ë'Z(X). So 
we use the alm.logous notation fiu quotients, ideals and associated graded alge- 
bras. For exami»le, A [zl'z means the graded algcbra of A with respect to the first 
filtrat ion. 
Definition B.1.  say that A is a topoiogicai fiitered alg«bra if the foltowing 
conditions are fulfilled: 
(1) A = ,,jez Mi'j' and the topolo on A is the strict inductive limit topology 
(this implies that A'd, A'' a.nd A , are closed in A). 
(2) For each i, j  Z the space A id is a nuclear Fr6çhet space. 
(3) The multiplication map it,j @ ia,ja  it+ia,j;l+ja is continuous with 
respect to the projective topology. 
(4) The canonical maps from A'd/A - ,J to the projective limit  k + -oe 
of A"¢/A d, and from A"J/A ''- to the projective limit as k + - of 
i,j/z.k, are isomorphisms of topological vector spaces. 
(5) The canonical maps ff'oin AI*I'J/A [i],- to the projective limit  k + -oe 
of AIiI'a/A IiI'e, and from A"[Jl/A -,[j] to the projective linfit  k + - 
of Mi'[J]/M t'bI, are isomorphisms of topological vector spaces. 
ç emphasize that our definition is less general tlmn the one considered in [2]. 
Since for fixed i, j the product map A »  A 'j  A is continuous, one defines 
the continuous Hochschild chain spaces as 
(35) c() ::  ,(+'). 
Out innovation consists in introducing ttree filtrations on C(A), in the following 
i,j;l 
way: a pure tensor a = ao ...  ak is said to belong to the liuear space c (A) if 
ao  N i°'j° .... , ak  N i'j with i0 + ... + ik  i, j0 + --  + j  j, and such that 
for any subset S C {0 .... ,k}, esi  ! and esJ  l. The spa.ce C'Jz(A) 
i,j ;l 
is defined as the closure of c (M) inside Ck(M), endowed with the inductive limit 
topology. One has the obvious inclusions A j'j(k+l) C Ck+lD'(k+l)J:(k+l)J(A) and 
C'J;(A) C A'(+l); thus 
c()= U c'»'(A)  
«j.lZ 
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Since the map b is continuous, it extends to the t.opological chain spaces. The 
topological Hochschild homology of .A is then defined as the homology of the conl- 
 j- 
plex (C(A), b). The advantage of the tri-filtered spaces Q " (A) is that 
':(A), 
while no analogous property holds for J 
Since the filtrations are preserved by b, we tan form spectral sequences for the 
homology of the Hochschild complex with respect to those filtrations. A fundamen- 
rai diculty appears when analyzing suçh spectral sequences: the E ° terres live 
on a half-plane. Thus. tauologi«al properties {}f first-qnadrant spectral sequences 
(likc collapsing or convergence) will need carefifl proofs. In particular, we will need 
Io smn infinitc series of chains. 
'om ri)c, dcfinition of t()l)ological filtered alget)ras, every smn q=oaq, aq 
M t-q'k is a.symptoHcally summabl« in M 'k in the following sense: there exists 
u  A#'k such that fl)r all I > 0 w(" bave 
I 
q=l) 
The Salue lm)I)('rty also hol(ls fi)r the secon(l filtration. This prot)erty implies that 
i-q,j;l 
ev(,rv sequ(,nce aq  C. (), q OE (1. 
i" stress that this stat('nent, wonld fait if we let th(' third index / vary with q. 
N()w let ,4 'z i)e a topological filtered algebra such tha( t-eMk'z = 0. Let 
CE(A) denote the spectral sequence of the tlochschild complex of.A with respect to 
the first filtration. % say that cE(M) degenerates at CEP if the higher differentials 
dP, d p+l, etc. ail vanish. Th(,re exist na/ural filtrations on Hochschild homolo 
induced from the filtrations of the Hochs«hild complex: namely, a homology cls 
lires in the first filtration i if it can be represented by a cycle in C'Z;z(A). Moreover. 
there exist natural edge maps 
 P 
(36) HHi+q()[il  Ei.q(M) 
ri'oin the gra(led group of HHi+q(M) with respect to the filtration into the spectral 
sequence. % say that CE(A) converges (to the graded Hochschild homolo-) if it 
degenerates at cEP for some p, if the edge maps (to oç) are isomorphislnS, and if 
moreover the "residnal'" homology iez HH(A)i vanishes. 
Let (C 'z;z, d) be any tri-filtered homology complex, i.e., C. = U.j,tez C j:g- To 
increase readability of the next paragraphs we will use the following conventions: 
the subscript  will be generally suppressed.  will work with chains in C, C [zl'z;z, 
C'z'[zl;z or Ç[Zl,[Zl;Z. and we will indicate the chain space we are working in by the 
filtration indices: for example, a  Çi,l:l denotes a chain in C z,[zl;z. If a  Çi,j:l, 
we will denote again by a it.s image in the complex Çi'[];/. 
Lemma B.2. For an spectral sequence associated fo a N-filtered homologg com.plez 
(C2, d) and for all p  N, the edge maps Hi+q(C[i] 
,,q are injective. 
Proof. The upper index in C2 denotes the filtration, which is assmned to be in- 
creasing. Let []  H+k(C) be represented bv a cycle a  C  Assume that 
-  i+k" 
the image of a in E vanishes. Then there exists a cls [i+-l]  E - 
such that dp-l[i+p_l] = [a]Ep-. This means t.hat there exists a representa- 
tire ,di+p-  C i+p- of [i+p_] such that  -d(/i+-) represents the null 
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class at E p-l. Inductively, construct ti+p_2 E Ci+p-2,... ,[3i  C i Sll('h that 
et -- d([i+p-1 n u ... n u [i) represents the lmll class at "E °. But this means that 
et - d(/3i+p-1 + ... + 13i)  C i-1, or lu other words that [ai e Hi+q(C)i-1, as 
clailned. [] 

This lemnla is proved using diagralu dmsing. The esseutial step in the proof is 
sunlming/3i+p-1 + ... + Pi iuside C i+p-1 As long as the smn is finite, there is of 
course no problem. However, diagram «hasiug for other iugredients of convergeuce 
will involve sumlning iufinite series of «hains of dcs«euding filtration orders. This is 
in prin«iple impossible, sin«e the other filtrations lnight go up So we llmSt almlyze 
each step of the diagram «hasing vcry carefully. 
Let N  Z be fixe& Ve assume that for any i,j  Z, j'  j, i'  i and 
l  1 + lnax(i + , j + d ), the following assmnptions hold tbr the COlnplex C (these 
will be checked individually for the ttochschild chain COlnplexes of the fibered CtlSp 
symbol algebrs): 
i.j;l that is exact modulo 
H1. For auy k ¢ i + N a.nd auy chain  
t.here exists fi  C i+'j+;t such that b(fi) + a  Çi,j+l:l and b(fl) + a is 
exact in Ç[i]j+i;l. 
id;t be such that b(a)  C i-2"j;l aud k ¢ i + N. Then  is exact 
H2. Letak 
modulo Çi-,;. 
H3. Let a  C i'j;l + C i''j-1;l be stlch that a is exact in C i'[j];l modulo C i-l'[j];l. 
Then there exists   C i+l'j;l such that b() + a  C i'j;z + C i''j-1;l and 
b() + a is exact in Ç[i],U];l. 
H4. Let a  Çi,j;l +Çi-l,j';l be such that a is exact in C [i]'j;l. Then there exists 
  C i';t such that b() +   Çi-l,j';l. 
tt5. Let a  Çi,j;l+ Çi',j-1;l + ci-l,j';l be such that a is exact in Ç[i],[j]:l. Then 
there exists   C id: Sllch that b() + a  C i-'y; + C i''j-1;l. 
- Ci'j;z for fixed j, k, ! is 
H6. iez Ci'J;t = 0, and auy series i=io 
asymptotically summable. 
The sumptions H1 and H2 sw slightly more than the fact that "E(C) degener- 
ates at "E 2 and that cE? = 0 for * ¢ N. Assulnptiol H3 is slighth" stronger thau 
the degeneracy of "E(C '[1;) at CE 2. Assumptiol H4 ilnproves the degeneracy 
of °E(A[zI,z) at °E1. Finally, H5 gives Ulfiform bouuds lu the third filtration for 
exact chains. h remark here that this condition and a statelnent analogous to the 
second part of Lelnlna B.7 lnust be checked in the proof of the convergence result 
g,j;l be 
Proposition B.a. Assume that H1-H6 hold for the complez C. Let a 
an exact chain modulo C i-l'Z;Z. If k ¢ i+ N, then there exists   C i+l'j+l;l 8ch 
that b() + a G C i-d:t. If k = i + N. lhen lhe same conclusion remains valid with 
possibly higher indices j, l. 
Pro@ Assume first that k ¢ i + N. By H1 there exists o G C i+'j+:t such that 
b(o) + a G C'o+;t and b(flo) + a is exact in C[q,J+;. By H4, choose 
such that b(flo + ) + a G C i-'+;t. Assume inductively that for 0  p  q - 1 
we have fouud p G C i-p+'j+;t and fie  C i-q+,j+t such that 
b(o + ... + - + ) +   C i-«'j+;t + C i-'j:t 
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The initial st, ep q = 1 was donc above. Since et valfi.she.s in C i-q'[j+l];l, it follows that 
b(130+...+/3q-1 +/3'q)+o is exact in C i-q'[j+q;t. Bv H3 there exist.s/3' E C i-q+l"j+l;t 
such that b(/30+...+/3q_+/3'q+/sq')+a  Ci-q'J+m+C -l'j; is exact in 
! 
By H5 thcrc exists 13q+  C i-q'j+l;I such that 
b(/ 0 4- 4-/3q_ 1 4- i t tt t . 
.... q +/3q +/3q+) + c  C-q-lJ+: + C i-° 
Sett.ing/3q := t3q +/3q , we complete our induction argulnent. 
By H6, the sel-ies 13q is asymptotically sunmmblc; so set /3 := q=0 
ci+l,j-t-i;! Since 
k+l  
d(/3) + c  C i-4: + Ç] c i--IJ+: = c 
q>0 
using H6, the COlclu.sion follows fol /," ¢ i 4- N. A.ssmne now lhat k -= i 4- N. The 
first step of thc argument goes through if we allow for po.ssibly higher .). . In the 
re.st of the proof we did hot invoke H I or H2. So evervthing goes through for these 
new indice.s j.l. [] 
There i.s .sOlnething very nontrivial about this prop«sitiol. Even without caring 
aboul the third index (which is almoying but hot sei'ious), a. simple application 
of III would yield a «hain /3  Çi+l,j+l;l .sll('h that d(/3) + c  C'-'¢+l:t Then 
diagram chasing to write n as a bounda,ry wouhl lead to an infinite series of chains 
of decreasing order in i, but increasm9 in j. It is fiwtunate that we can reduce 
back the second index iii the -case (mainly thanks to the fact that H3 holds in 
this ca.se), but we believe that for a nmlti-filwation pseudodifferential algebra this 
would hot be possible. 
Theorem B.4. Let C be a tri-flltered complez sati.sfgin9 assumptios H1 H6. Then 
the .spectral sequence cE(C) is convergett. 
 2 
Pro@ We would like to invert the edge lnaps (36). Bv H1. H2 and H4 the Ei, . tenn 
 2 
is COlmentrated on thc line  = N. Then let 4 bc a class at Ei. N. Then A can be 
ti°:t (note that in the case of the Hochschild complex 
l'epl'eSelled by a. chain or I  "--'i+N 
associated to 2"o. j tan be taken j := i + N. sec Theoreln 5.3). Surviving a.t °E 2 
r"[i-q'J:t is exact. Thus. bv H4 there 
Ci-l'j;I and that b(Cl)  "-i+N-1 
ilmans that b(c) G i+v- 
g--i--2,j:t Since (i+N-1)-(i-2) ¢ N 
,-.y;t such that b(c)+b(a)  "- i+V-l" 
exists a2  "-i+N 
and b(c + eq) is exact modulo anything, we can appl,v Proposition B.3 to get 
(,i-3,jl Inductivelv, using H4. 
Ii--l'j+l;I Sllch that b(eq 4- c2 + (t3)  "-i+N-l" 
(3  "'i+N 
(-i--q,j:! 
construct cq  t"i--q+2'j+l:l._.i+N Silch that b(cq + .... + q)  i+N--I Bv H6, the sure 
O := Zq=l  Ctq  t"i'j;l"-i+N is asylnptoticallv  smnlnable and represents ,4 at °E  bv 
construction, and 
b() e Ç] .+u,,Æ-.; _- 0. 
qGN 
So we have found a Hochschild cycle c that nlaps to A under the edge nlap (36), 
which lneans that the edge maps are isomorphisms (we have seen above that they 
are injective). 
Now we prove the vanishing of the "residual holnology" Ç]iez HH:(,A)i. In fact, 
we prove directly that HHk(al/[--q = 0. Let [ct] be a homology claus represented 
bv a cycle c G Çk-N-I,1;I Bv H2 and Proposition B.3. there exists/31 



HOMOLOGh OF FIBERED CUSP OPERATORS 3043 

such tllat b(13t) + c E C k-N-2"j;l, hlductively, there exists /3q  C k-N-q+l'3-t-1;l 
such that b(/31 +... +/3q) + a  Çk-N-q-l,j;l. The smu/3 := q=l/3q is summable 
iii C k-N'j+l:l, and nloreover 

b(/3) +   Ç] c "-N-q-l,j: = o. 
qN 

which shows that a is exact. 

Remark B.5. Prolosition B.3 remains valid if we allow anv llllIllber of exceptional 
N's in H1 and H2, with tlle obvious modifications. Thns, Theorem B.4 holds if 
we allow a finite nUlldWr of sueh exceptional N's. This is needed, for instance, foi" 
convergence in the case of the adiabatic algebra [35]. 

We are now going to establish some sufficient conditions for a topological filtered 
algebra and the associated symbolic spectral sequenee to satisfv the conditions H3. 
H4, and H5. So for the test of this section we assmn« that I], is isomorphic as an 
algel,ra to C[p, p-]  A, where ,I is one of the algebras £  AI01,[0] or C(Y)[x-], 
for } a compact manifold with bomularv and .r a bomMary-definiug flmction for 

Lennna B.6. Properties H4 and H5 are .satisfied by ,4 under the above assumption. 

Pro@ The algeln'a A ['l,[z] is isomorphic to the tensor product A [°l'[°] ®C[p, p-] ® 
C[a',.r-]. Bv the Eilenberg-Zilber theoreln, we tan l'eplaee tlle H¢chsçhild com- 
plex of A ll'['] by the tensor product of the H,chsçhild complexes C.(.,_lI0],[0]) ® 
C.(C[p,p-]) ® U.(C[a',x-]), since thc explicit quasi-isomorphisms between the 
two complexes (the Alexander-Whitnev map and the shuffle product) preserve the 
two gradings and the third filtration index l. It is then enough to prove the state- 
ment for the algebra C[a', ac-] itself. This is done bv writing ail explicit homotopy 
between t he Hochschild complex C. (Clin, ac- 1]) and t he so-called small complex (see 
[20. Section 3.1]). Hence we get H.5. Property H4 is proved in exactly the smne 
way, bv using forlnal Lam'ent series rather than Laurent polynomials in the vm'iable 
x -. [] 

Note now that the a.lgebra A [z]'z and hence also HH.(A [l'x) have two gradings 
(depending on choices), given bv the negative of the total degrees in p and x. 

Lemma B.7.  Let [c]  °E.k(A: [z]) be a class that corresponds to a class of 
degree (i,j) when seen as an element of HH+(A[z]'[z]). Then there exists a  
ci,j;max(i+$,j+$y) (A2.[z]) presenting lai 
i+k  
 Let lai  ,tA   be a class that corsponds to a class 4de9ree (i,j) when 
s««n as a, «l«,n«nt 4 HHi+(A[z]'z)  Thon th«ç« «zists ce C:Ç('+°'J+)(A :'[z]) 
representing 

Pro@ It is enough to prove the statelnent for the algebras Çee(Y)[x-1], £ and 
C[p,p -] (which is obvious), and then to use the shuffle map to the Hochschild 
complex of the tensor product algebra. [] 

Corollary B.8. Assume that the spectral seqence «E(A ,[z]) degenerates at cE 2 
Then condition H3 is fulfilled. 
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Pro@ Let o E C 'j;z be such that ( is exact in C i'[j];l lnodtllo C i-l'[j];l. This 
mea.ns there exists fl E C i+p'j;z such that b(/3) + c  C -l'[al;z. Then c survives 
as the lmll class in «/p(7,[Z]), sO bv hypothesis also at CE 2. In other words, there 
exists []  cE 1 wth dl[] = -[c]. Clem'ly [] is of (negative) degree (i + 1,j) 
in p and x. By Lemma B.7 we can choose a representative/3  C i+l'[];t provided 
l _> max(i + 15,j + 5). This is the desired chain ff. [] 

Corollary B.9. Let A be a topological flltered algebra such that the second filtration 
is co'nstant. Assume that cE(A) d«g«nerates at CE , that A [z] - Cp,/9 -1] @ .4 [°1, 
and thal M [°l is H-umlal. Then cE(M) is convergent. 

Because the gradcd algcbra is a tensor product, we see bv LelnlnaS B.6 and B.7 
that if a  C'i;z is exact modulo C'i-;z, then there exists ç E C i+; such that 
b(fl) + c  C'i-:, l)rovided I _> i + I + 5ï. This allows us to conchlde that the edge 
nlaps are isoln(»rl)hisllls and t|la.t t|le residual homology vmlishes, by constructing 
asynlptotically smmnablc serics of chains via diagram chasing as iii Theorem B.4. 

The last result is ilnplicitly used in [4]. 

1 EFERENCES 

[1] B. Ammann, R. Lauter, V. Nistor, and A. Vasy. Complex powers and non-compact manifolds, 
mat h.OA/0211305, preprint, November 2002. 
[2] M. Benameur and V. Nistor, Homology of complete symbols and noncommutative geometry, 
Landsman. N. P. (ed.) et al., Quantization of singular symplectic quotients, Basel, Birkhguser. 
Prog. Math. 198 (2001), 21-46. 
[3] J.-L. Brylinski, A dzfferential complex for Pozsson manifolds, J. Differential Geometry 28 
(1988), 93-114. MR 89m:58006 
[4] J.-L. Brylinski and E. Getzler, The homology of algebras of pseudodifferential operators and 
the noncommutative residue, K-Theory 1 (1987), 385-403. MR 89j:58135 
[5] B. Bucicovschi, An extension of the work of V. Guillemin on complex powers and zeta func- 
tions of elliptic pseudodifferential operators, Proc. Amer. Math. Soc. 127 (1999), 3081-3090. 
hIR 2000a:58085 
[6] A. Connes, Noncommutatzve Geometry, Academic Press, New York - London (1994). MR 
95j:46063 
[7] A. Connes, Gravity coupled with matter and the foundation of non-commutative geometry, 
Comm. Math. Phys. 182 (1996), 155-176. MR 98f:58024 
[8] H. O. Cordes, On a class ofC*-algebras, Math. Annalen 170 (1967), 283-313. MR 35:749 
[9] Yu. V. Egorov and B.-W Schulze, Pseudodifferential operators, Singularities, Applications, 
volume 93 of Operator Theory and Applications. Birkguser, Basel (1997). MR 98e:35181 
[10] C. L. Epstein, R. B. Melrose, and G. A. Mendoza, Resolvent of the Laplaczan on strictly 
pseudoconvex domains, Acta Math. 167 (1991), 1-106. MR 92i:32016 
[11] B. Gramsch. Relative Inversion in der StSrungstheorie von Operatoren und k9-Algebren, 
Math. Annalen 269 (1984), 27-71. MR 86j:47065 
[12] V. Guillemin, A new proof of Weyl's formula on the asymptotic distribution of eigenvalues, 
Adv. in Math. 55 (1985), 131-160. MR 86i:58135 
[13] R. Lauter and S. Moroianu. An index formula on manifolds with flbered cusp ends, preprint, 
2002. 
[14] R. Lauter and S. Moroianu. Homology of pseudo-differential operators on manifolds with 
fibered boudaries, Journal Reine Angew. Math. 547 (2002), 207-234. 



HO/klOLOGY OF FIBERED CUSP OPERATORS 31)45 

[15] R. Lauter and S. ikloroianu, The index of cusp operators on manifolds with corners, Ann. 
Global Anal. Geom. 21. nf. 1 (2002), 31 49. MR 2003e:58033 
[16] R. Lauter and S. lloroianu, On the index formula of IkIelrose and Nistor. Preprint Nr. 3, 
IMAR, Bucharest, lklarch 2000. 
[17] R. Lauter and S. Moroialm, Fredholm theory for degeneratc pseudodifgferential operators on 
raanifolds with Jïbered boundaries, Comm. Partial 13ifferential Equations 26 (2001), 233-283. 
MR 2002e:58052 
[18] R. Lauter and V. Nistor, Aalysis of .qeoraetric operators on open raanifolds: a .qroupozd ap- 
proach, In N.P. Landsman, M. Pflaum, and M. Schlichenmaier, ed., Quantization of Sin.qular 
Syraplectic Quotients, vol. 198 of Pro.qress in Matheraatics, pp. 181-229. Birkh/user, Basel - 
Boston- Berlin, 2001. 
[19] M. Lesch and M..l. Pflaum, Traces on al.qebras of pararaeter dependent pseudodifgferen- 
tial operators and the eta-invariant, Trans. Amer. Math. Soc. 352 (2000), 4911-4936. iklR 
2001b:58042 
[20] J.-L. Loday, Cyclic homology, volume 301 of Grundlehren der Mathematischen Wis- 
senschaften, Springer-Verlag, Berlin- Heidelberg- New York (1992). MR 94a:19004 
[21] F. Mantlik, Norrn closure and extension of the syrabolic calculus for the cone al.qebra, Ann. 
Global Anal. and Geometry 13 (1995), 339-376. MR 97a:58183 
[22] R. R. Mazzeo, Elliptic theory of difgferential ed.qe operators I, Comm. Partial Differential 
Equations 16 (1991), 1615-1664. MR 93d:58152 
[23] R. R. Mazzeo and R. 13. lklclrose, Pseudodifgferential operators on raanifolds with fibred bound- 
aries, Asian .l. Math. 2 (19.q8), 833 866. IIR 2000m:580-16 
[24] ,I. McCleary, User's .quide to spectral sequences, volume 12 of Nlathematical Lecture Series, 
Publish or Perish, Wilmington (1985). MR 87f:55014 
[25] R. 13. Melrose, Analysis on raanifolds with corners, in preparation. 
[261 R. B. Melrose, Pseudodijïferential operators, corners and singular liraits, In Proceeding of 
the International Congress of Mathematicians, Kyoto, Springer-Verlag, 13erlin - Heidelberg - 
New York (1990), 217-234. NIR 93j:58131 
[27] R. B. Melrose, The Atiyah-Patodi-Singer zndex theorera, volume 4 of Research Notes in 
Mathematics, A. K. Peters, Wellesley, Massachusetts (1993). MR 96g:58180 
[28] R. 13. Melrose, Spectral and scattering theory for the Laplacian on asyraptotically Euclidean 
space, In M. Ikawa (ed.), Spectral and Scattering Theory, volume 162 of Lecture Notes in Pure 
and Applied Mathematics, pages 85-130, New York, 1994. Marcel Dekker Inc. Proceedings 
of the Taniguchi International Vorkshop held in Sanda, November 1992. MR 951(:58168 
[29] R. 13. lklelrose, The eta invariant and farailies of pseudodifgferential operators, Math. Res. 
Letters 2 (1995), 541-561. lklR 96h:58169 
[30] R. 13. Melrose, Geometric scattering theory, Cambridge University Press, 1995. MR 
96k:35129 
[31] R. 13. Melrose, Fibrations, compactifications and algebras of pseudodif:ferentzal operators, 
In L. HSrmander and A. Mellin. editors, Partial Differential Equations and Mathematical 
Physics, Birkh/iuser, 13oston, 1996, 246-261. MR 98j:58117 
[32] R. 13. Melrose, Geometric optics and the bottora of the spectrura, In F. Colombini and 
N. Lerner, editors, Geometrical optics and related topics, volume 32 of Progress in nonlinear 
differential equations and their applicalions, Birkh/iuser, 13asel - Boston - 13erlin (1997). 
[33] R. 13. Melrose and V. Nistor. Higher index and -invariants for suspended algebras of pseu- 
dodifgferential operators, unfinished manuscript. 
[34] R. B. Melrose and V. Nistor, Horaology of pseudodifferential operators I. Manifolds with 
boundary, to appear in Amer. Malh. J., Preprint, lkloEv 1996. 
[35] S. Moroianu, Higher residues on the algebra of adiabatic pseudodifgferential operators, Ph.D. 
thesis, Massachusetts Institute of Technology, Cambridge, Massachusetts (1999). 
[36] W. Millier, Manifolds with cusps of tank one. Spectral theory and L 2-index theorera, Lecture 
Notes in Mathematics 1244. Springer-Verlag, Berlin (1987). IIR 89g:58196 
[37] R. Nest and E. Schrohe, Hochschild homology of Bouter de Monvel's algebra, in preparation. 
[38] T. M. W. Nye and M. A. Singer, An L2-index theorera for Dirac operators on S 1 × 13, j. 
Funct. Anal. 177 (2000), 203-218. lklR 2002a:58020 
[391 V. Nistor, Groupoids and the integration of Lie algebroids, J. Math. Soc. Japan 52 (2000), 
847 868. iklR 2002e:58035 



3046 ROBERT LAUTER AND SERGIU MOROIANU 

[40] S. Rempel and B.-W. Schulze, Complete Mellzn and Green symbohc calculus zn spaces wzth 
conormal asymptotics, Ann. Global Anal. Geonmtry 4 (1986), 137-224. MR 89f:58132 
[41] C. E. Rickart, Banach algebras with an adjoint operation, Ann. NIath. 47 (1946), 528-550. 
hIR 8:159b 
[42] B. Vaillant, Index- and spectral theory for manzfolds with generalzzed Jïbered cusps, Ph.D. 
thesis, University of Bonn (2(101). 
[43] hl. \Vodzicki, Cyclic homology o,[ differential operators, Duke hiath. J. 54 (1987), 641-647. 
NIR 88k:32035 
[44] h|. Wodzicki, Noncommutative residue. I. Fundamentals, In K-theor,: arithmetic and geom- 
etry (Moscow, 1984-1.q86), Springer, Berlin (1987), 320-399. hlR 90a:58175 
[45] h. \Vodzicki, Cyclic homology of pseudodiferential operators and noncommutative Euler 
class, C. R. Acad. Sci. Paris Sér. I Math. 306 (198a), 321-325. hlR 89h:58189 
[46] J. V'unsch. Propagation of singularities and growth for Schrdznger operators. Duke h|ath. 
J. 98 (1999), 137-186. hll 2000h:58054 

FACHBEREICH 17 - hI.THEMATIK, UNIVERSIT.T h|AINZ, D-55099 IAINZ, GERMANh 
E-mail address: lauter@mathematik.uni-mainz.de 

INSTITUTUL DE [ATEMATICÂ AL A('ADEr, IE Rot« NE. P.(). Box 1-764 RO-70700 Bi CHAREST. 
[()MANIA 
E- mad addr¢ss: moroianu@alum, mit. edu 



TRANSACTIONS OF THE 
AMERICAN MATHEMATICAL SOCIETY 
Volume 355, Number 8, Pages 3047-3064 
S 0002-9947(01 )02893-8 
Article electronically published on September 21, 2001 

MODERATE DEVIATION PRINCIPLES 
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ABSTRACT. Let {Xn} be a sequence of i.i.d, random vectors with values in a 
separable Banach space. Moderate deviation principles for trajectories of sums 
of {Xn } are proved, which generalize related results of Borovkov and Mogulskii 
(1980) and Deshayes and Picard (1979). As an application, functional laws of 
the iterated logarit hm are given. The paper also contains concluding reinarks, 
with examples, on extending resulls for parlial sums to corresponding ones for 
trajectory setting. 

1. INTRODUCTION AND MAIN RESULTS 

Let {Xn} be a sequence of i.i.d. R'Cvalued random variables, satisfying EX1 = 0 
and Var(X1) < +oe. Let çn be the trajectories of sums of {X,}, that is, ,(t) = 
 X + (nt- [ntl)X[l+l,t  [0, ll. Deshayes and Picard [191 have studied moder- 
i=1 
are deviations (MDs) for {ç, } in C[0, 1], which generalized corresponding results 
obtained by Borovkov [10] and Mogulskii [301. Bo'ovkov a,,d Mogulskii [12] ex- 
tended Deshayes and Picard's [19] results to independent, identically distributed 
(i.i.d.) random vectors {X} with values in a complete locally convex Hausdorff 
topological vector space, under the crucial assumption that 

I. = £(X)  CLT. 

bution. For more related results, see Borovkov and Mogulskii [12] and the references 
therein. However, the CLT for i.i.d, random vectors {X} is hot easily satisfied 
when {X.} take values in a general Banach space. Motivated by the observation 
above, in the present paper, we shall investigate the moderate deviation principle 
(MDP) for the trajectories, S., of sums of i.i.d, random vectors {X} with values 
in a separable Banach space, and aire dt removing the assumption emplo.ved bv 
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Borovkov and Mogulskii [12], p = £(Xl) E CLT. As ail application, t]le flnct.ional 
laws of the iterated logarithm are given. 
Now, we turn to describillg our main results iii detail. Let N be the set of 
positive integers. For   R, [x] denotes t.he greatest integer k  x. Through- 
out this paper, (E, Il" Il) wilt denote a separable Banach space and E* its dual 
space. For N  N, we endow the product space E N with the product topoloD', 
N 
which Call be induced by the lnetric d(-,-) given by d(x,y) =  I,- ll, = 
(x,... ,xN),y = (Yl,"" ,Y)  E N- Let {X} l)e a sequence of independent E- 
valued randoln vectors with COlllnlOll distribution p such that p  ll'Al, that is, 
f(X1) = 0 311(1 f2(«¥1) < OE for every f e E*. Let (H, Il- IIH) ge the reproducing 
kerllel Hilbcrt space associatcd to # (sec Goo(hllall, Içuelbs and Zillil [22]). A good 
exalnple, which reveals the structure, is the Wiener lneasure p Oll E = CI0, 1] with 
the sllprelllllnl llOrlll. Thell file associated reproducing kernel Hilbert space is the 
so-callcd Canleron-Martin st)a.«e. Let S, =  Xi, Sa = 0.  dellOte bv n the 
i=1 
traje('tories of 8111118 of {Xn}. hl other words, 
.ç..(t) = S[t] + (rit - [,,t])X[t]+,. t C [0.1]. 
Denote by Sa(-) the piecewise ('onstallt fllIICtiOllS of sIlnlS of {Xn}, that is, 
s(t) =  x, t e [0, 1. 
Let C([0, 1],E) be the Bana«h spe of ail «ontinuous mappings from [0,1] into E 
equit)ped with the sup-norm, [. []. Denote by D([0, 1].E) the Bana«h spa«e of 
all right-continuous mappings with left limits fronl [0, 1] into E equipped with the 
metric d(f,g)= sup f(t)-g(t)]]. Given a set 4. let A c and .4 stand for the 
0<t<l 
complement and the closure of A, respectively. 
Define a flmction A  E  [0, 
{, 
(].1) A() = 5llxll, if  e , 
+., ot}lervise, 

and define a nmpping 
(1.2) ï(f)-=- { +oc, f) A(g(s))ds, 

f EAC, 
ot herwise, 
where AC = {f e D([0,1], E); there exists 9 e Lt([0,1], E) such that f(t) = 
f 9(s)ds for t e [0, 1]I, and L([0, I],E)is the spa«e of E-valued Bochner integrable 
functions on [0,1]. It is readily seen that AC c C([0, 11, E). 
Throughout this paper, for randoln vectors {I}, we will write I  0 if 
  0 in probability    oe. 
Throughout this paper, let {b(n): n  1} be a positive sequence such that 
(1.3) b(n) b() 
The following conditions are sumed to be satisfied. 
(1.4) Eexp(llXll) <  for some  > O. 
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-zt P 
(1.5) b(n)  0. 
It is well-known that under conditions (1.4) and (1.5), {  } in E satisfies 
gll 
MDP with speed {} and the rate function A, defined by (1.1). By this 
we 
mean that, %r every closed set F C E, 
(1.6) limsuplogP F - inf A(x), 
n xF 
and for every open set G C E, 
(1.ï) liminflogP e a OE - inf A(z). 
noe xG 
Throughout this article, a rate fimction is undcrstood to bave compact lcvel sets, 
i.e., {A  a} is compact in E for all a k 0. For this result of the partial sure 
see Goodman, Kuelbs and Zinn [22, Lcmma 2.1 (V)], Chen [13, Th«*orcm 2]; [15, 
It should be mcntioncd that rcccntly Arconcs [7] gave necessary and sufficient 
conditions for MP for 
The main results of this aper are following. 
Theorem 1.1. Let'O=to < ¢l <'"< t = l beapaiioof [0.1]. Assume 
wih speed  ad a rae fucio I , ha is, for ever closed sec F ad 
ope se G of E , 
n { 1 } 
limsuplogP (Sn(t,),'",Sn(tN))F <-infÏ(z) 
n { 1 } 
where Ï - E [0,+oe] is 9ive b 
N 
for z = (zl,--',zN)  E N, zo :0. 
So does { (n(t,),'" ",n(/N))} in E N with he saine speed {  } and 
saine race fuc¢io I . 
Remark 1.1. It is readily checked that Ï  inherits the proerty of compact level 
sers from function A. 
Theorem 1.2. Assume (1.d)ad (1.,)hold, he {} i Ç([0,1],E) 
an MDP with speed {  } and rate function À d:ned Vy :1.2), that is, for every 
closed set F and open set G of C([0, 1], E), 
limsup  lOgPn+  e F  - IeFinf 
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liminf,__,  logP b C G -> - gec;inf À(g). 
[ b() 
flmct¢onÂ. 
Remark 1.2. From the proofs given in the next section, it can be seen that all 
lenmms, and thus Thcorems 1.1 and 1.2 remain truc if we substitute the underlying 
intcrval [0, 1] with intcrval [0, T] for each T > 0. 
Remark 1.3. Theorem 1.2 has generalized the relatcd results obtained by Borovkov 
and Mogulskii [12] and Deshaves and Picard [19]. 
If E is of type p (1 < 1' < 2), th«,l condition E]X ]]P < oe implies   0 (sec 
nl/p 
Lcdoux and Talagrand [27, pp. 1911 and 259]). Thereforc, an immediate corollarv 
of Thenrem 1.2 is following. 
Corollary 1.1. Sut,pose E is of type V, 1 < p < 2. and Exp(fl]]Xl]]) ( OE for 
rate flmction Â defirwd b 9 (1.2). 
It should be mentioned that the large deviation principles (LDPs) for {;   1} 
have been studied extensively, sec Varadhan [32], Borovkov [10], Mogulskii [30]. 
Deshayes and Picm'd [19], Borovkov and Mogulskii [12], Schuette [31], Dembo and 
Zajie [16], Hu [23] and the references therein. LDPs for sample paths of vector- 
valued Lévy processes have also been proved by de Acost.a [3]. Arcones [7] gave 
necessary and sucient conditions for LDPs and MDPs for {S,,(.); n R 1} in the 
real-valued case. 
The projective-limit met hod has been developed to successfiflly t reat nmny prob- 
lems. For a nice account of the theory, sec, for exmnple, Dembo and Zeitouni [18]. 
 do hot sec how to prove our result bv the projective-limit method. A method 
of subsequences is devised to establish the MDP upper bound. The lower bound 
is proved by an interesting cal«ulation which is rather explicit. Both upper and 
lower bounds are presented in Section 2. Our method is also different from that of 
Borovkov and Mogulskii [12]. It wouht be interesting to sec our problem worked 
out along the line of the projective-limit lnethod: comparison of the two approaches 
should be instructive. On the other hand, it should also be interesting to look on 
the MDPs for stochastic processes in general, sec Arcones [7] for corresponding 
results in real-valued case in this direction. 
The Functional Laws of the Iterated Logarithm will be considered in Section 3. 
In Section 4, some remarks will be given. 

2. THE PROOFS OF MAIN RESULTS 

We begin with severa.1 lemmas, which are important for proving the main results. 

Lemma 2.1. Given e > O, 0 < 6 <_ 1, under condition (1.5), for all sufficiently 
large . we bave 
P{l<k<[n6]+lmax ,]Sk]l > ïb(T/)} < 2P{"S[n6]+l'] > 
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r ,,,x Il&Il m ch(,,) <  Il&il > b(,,) . 
l<k<n  
Pro@ We will prove only the first iucquality, for the second cau bc trcatcd similarly. 
Bv Ottnvia.ni's inequality (sec also Araujo and Gine, [6. pp. 110-111]) for every 
A > 0 alld OVel'V integer AI  1, 
A = b('n). By (1.5), we Cml steadily prove that for sucientlv 

Ta.ke /1I = [ma], 
large n, 

Lemma 2.2. Uder condition (1.,), {} aT«/ 
equivalent in D([0, l]. E). thal is. for ea«h ( > O. 

lin sup b log P ,,--, do b(n)'b(n) 

b(n) j are expone'ntia.lly 

> } = 

each a > O. 

<_ naXl<k<n IIxll, b,-Chebyshev's inequality, for 

_< ,,'{llXlll > 
<_ n exp(-/3ab(n))E exp(/lIX, II), 

where/ > 0 is as in (1.4), ri'oto whieh the expo,lential equivalence of ¢  "(') }  and 
{ s,,(-) l, follows. Lenmm 2.2 is proved. 
b(n) j 

Following the proofs of Chen [13, Theorem 2]; [15, Theorem 1]), one can steadily 
prove the following version, Lemma 2.3, of the MDP for subsequences of i.i.d, sums. 

Lemma 2.3. Let {t)} be a sequence of i.i.d, random variables with values in 
(E, II" Il), with conmon distribution p  ll'3Iô a'nd such that Eexp(cdl>]ll)< ce 
for some a > O. Let {nk} and {a} be positive integers and positive numbers, re- 
spectively, such that as k --, c,c, 

ak 

k 
where S,,, =  }), k >_ 1. 
j=l 

ak 

Sn k P 
ak 

 and the rate function I(x) = 
A(z), where A is deflned bg (1.1). 
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Lemma 2.4. For 0 <_ s < t <_ 1 fixed, let ||(s, t) = Sn(t) - S(s). Assume (1.4) 
and (1.5) hold. then b(.) in E satisfies an MDP with speed (n) and a 
rate functzon Is,t, that is, for every closed set F and open set G of E, 

liiusup n 
k-  log p b(n) e F 

<_ - inf 
xEF 

where l.  

(2. L) 

liminf n { lIÇ(s,t) 
k--. b-- l°gP b(,) 

G, > - inf ls ,(x), 

E -- [0, oe] is dcfined as follows 

( .r "] 1 A(x), xE. 
,,() = (t - )h ] - t . 

Proof of Lemma 2.4. The function I». dcfincd by (2.1) bas compact level sets be- 
cause A docs. sec Good,,m,,, Kuclbs and Zilm [22, Leln,na 2.1(V)] al,d Chen [13. 
Thcorcm 2]; [15, Theorem 1]. 
Given 0  s < t  1. defim' ,, = [kt]- [ks] and a = b(k), k  1. Then li (s,t) 
l  (s,t) 
has the saine distributi<m as S,+ and hence g( h the saine distribution  
. From (1.5) and Lemma 2.1 it follows that   0. Taking into account the 
ak ak 
fact tlmt       k  oe, Lemma 2.4 follows from Lemnm 2.3. Lemma 
2. is proved. 

Proof of Theorem I.I. Given a partition, n, of [0, 1],n : 0 = t0 < ri < "'" < tN = 1, 
denote IVn  = (S,(t),S,(t2)- S,(t),...,S,(tir)- S(tlv-)). Then by Lemma 
2.4. Lvnch and Sethuralnan [28, Çorollary 2.9] (see also Dembo and Zeitomfi [18, 
Ex. 4.2.7]), ,(__)7_b() -/ satisfies an lkIDP ill E N with speed { 6-) } and rate function 
I' giveu by 
N 
+=1 ti -- ti-1 
for z = (Zl," ",zu) 6 E u . 
Applying the contraction principle (see Delubo and Zeitouni [18, Theorem 4.2.1]) 
N 
to the continuous one-to-one map (Zl,"" ", ZN) -- (Zl, 21 -1- Z2,'" ", E Zi) on E N, 
i=I 
J" w it follows that b- (S,(tt), -, 
froln the MDP for the sequence 
satisfies an MDP in E N with speed (--)  and rate function defined by the 
equal- 
ity (1.8). Lemma 2.2 ilnplies that the sequences { 
b--(S,(t),..-,S(t))} and 
(& (,(tl),''',,(tN))} in EN are exponentiMly equivalent. Hence. bv Dembo 
and Zeitouni [18, Theorenl 4.2.13], (& (.(tl),''',n(tN))} 
in E/V satisfies the 
MDP as that for  ; (S,(t),-- S,(tv)) Theorem 1.1 is 
saine 
proved. 

Proof of Theorem 1.2. The compactness of level sets of À will be proved in Appen- 
dix B. 
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Upper bound. We first show tllat, for every f E C'([O, 1], E), and every e > 0, 
there exists a ba.ll of f, t3(f,p) --- {g; []f - g[[oe < p} for some p = p(f,e) > O, s,,ch 
that 
(2.2) lira sup .-572N_  log P 
To this end, given f  C([0, 1].E) and e > 0, we first consider the case where 
,(f) < 
For every partiîion, n, of [0, 1], n : (1 = t0 < ri <    < iN -= 1, by Theorem 1.1, 

(2.3) 

limsup  logPn__,oe b-  t3(f,p) 
-< limsup b-- l°gP,. b(n-- e Bo(I()),i= 1....,N 
<_ --illf {Ï-F(z); z = (Zl,"" ",zlv),zi G 13p(f(ti)),i = 1,...,1¥} 

where p(.I') -= { E E; Hx -- /[] < 1)} fol" 3"  E, and Ù(-) is as il, (1.8). \Ve cal, 
choose partition 7r such that 
(2.4) Ù((f(t),.. ",f(tN))) > À(f) -- ¢/2. 
Bv the lower senficontimfity of I F, we can choosc p = p(f, ¢) > 0 small enough 
such that 
(2.5) illf 
 ?((f(t),...,f(t)))-e/2. 
So (2.2) follows from (2.3)-(2.5). If À(f) = +oe, then we can sinfilarly prove that 
(2.2) is still true. 
Keeping (2.2) in mind, by a well-known standard argument we know that the 
upper bound holds for compact sets. Therefore, in order to complete the upper 
bound, it suffices to prove that -  "()[ - is exponentiallv tight, that is, for 
everv 
L > 0, there exists a compact set KL C C([0, 1], E), such that 
-t' " {(') "«} b( 
(2.6) i,,lSpog z  -. 
 will adapt Dembo and Zajic [16]'s argulnent to prove (2.6). However, a much 
more delicate est,mate is needed when we prove (2.7) below. 
By Theorem A in the Appendix A it suffices to prove 
 is expouentially tight, that is, for each 
a > 0, there exists a compact set h Ç E such that 
lim 
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(ii) For each p > 0, 
(2.7) liln liln sup ' {lsuPl[n(t)--n(s)">p}=--oc. 
6-0 ,-_.oe  logP b(n) it_sl<6 -- 
To this end. (i) follows froln Lemma 2.4 and Lvnch and Sethuraman [28. Lemma 
To prove (2.7). Noting Lemma 2.2, it is enough to prove 

t" 
(2.8) lira lilnsup u / 1 
-0 ,_-,¢¢ b2-n) log P bot) It-l<sup 
for each p > 0. 
In fact, fixp >0, for u _> 1 and 5>0, 

If-si<6 -- O<k<n,NraN[nS]+l I IIi 3=k÷ 

Noting 

for some k,O <_ k <_ n, and some m, 1 _< m <_ [] + 1. Then there exists a unique 
integer p, 0 _< p _< l - 1, such that 
([1 + )P -<  < (["l + )(p + 1). 
Hence, there are two possibilities for k + m. One possibility is that 
([nS]+l)pk+m< ([nS]+l)(p+l), 
in which case either 
The second possibility is that 
([l + )(P+ 1)   +,, < ([l + )(P+ 2). 
in which case either [[Sk+ -S([61+(p+) 
b(n.)p, or [[Sk - S([6]+)p[[ R b(n)p. In conclusion, we see that 
Okn,lNmN[n6]+l j=k+l 
p=o - - =p([n,]+l)+l 

P max Z X > b(n) 
l <m<[n6]+l 
-- -- j=p([n]÷l)÷l 

For () <  _< 1, let 1 =/() >_ 2 l,e the ulfique integer satis5ing 1/! <  <_ 1/(1 - 1). 
Then u < ([tt(] + 1)/ for suffici(,ntlv large t. For such a large n. suppose 
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by (2.9), (2.10) aIld Lenlma 2.1, 
{ 1 sup 
(2 11) P b(n)If-sic5 

l 

b( n ) lllax X j 
_<_<[ns]+ 

{ 1 
_< 2(/+ 1)P  

<2(l+l)p{]]S[n6]]] > P)+2(/+I)p{]]X1]] > P} 
- b(n) - 12 b(n) - 12 " 
Bv (2.11), (1.4), Lemma 2.4 and Chebyshev's inequality, for all 0 < 5 _< 1, 

(2.12) 

< max 

1 

Note that for x E H, 

(2.13) I1:11 _< «11:11 

Lower bound. To prove the lower bound, it suffices to prove that for each piece- 
wisc linear function f E C([O, 1], E), and each p > O, 

(2.14) lim_+ieef b2- log P B(f,p) >-7(f). 
(See also Schuette [31], Hu [23], etc.) 
This can be easily reduced to proving for the case of a linear fimction f : t -- 
tx, t  [0, 1], x G E (see also Borovkov and Mogulskii [12]). No, we now focus on the 
case f(t) = tx, t  [0, 1], where x E E is arbitrarily fixed. 
For 0 > 0, x e E, let No(x) = {g  C([0, 1],E); IIg(1 ) - xii < 0}. Note that for 
p>O,O>O, 

.No(x) ç B(f,p) = .No(x)\{.No(x)\B(f,p)} C B(f,p). 
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I (1) } yields 
Since the MDP for the partial SlllllS I.  
linnLileef  log P  e No(x) 

,' 
> liileef  log P b(r) x < 0 

_> -A(x) = -À(f), 
in order t.o prove (2.14), it sutfices to prove 

(2.15) liln sui) v.---ffTZS  log P 
for a certain 0 > O. 
A little thought revcals that ail sucicntly small 
B(f. p) is a closed sct. we apl)ly lhe uI)pcr 1)omul result and estimate 
-(L. H. S. ()f (2.15)) 
O<t<l 
= inf inf{A(g);]]g(t)-t(x + z)ll  p-o for some 0 < t < 1,g(1) =  + z} 
Ilzll0 
 inf inf inf{Â(g);y=g(t)-t(+z),l[yllp-o.g(1)=«+z} 
iizlloo<t<l 
 inf inf inf {,A((+z)+ 9) ( 9 )} 
IlzllOO<t<l Ilvllp-0  + (1 - t)A (x + z) 1 - t 
> inf inf inf 1 y 2 y 
H 1--t H 
> inf inf inf 1{ 
 inf {1 
whose lilnit  0  0 is greater than or equal to A(z) + 2 () by the lower senti- 
contilmity of the rate function A, where « is as in (2.13). 
This concludes that (2.15) indeed holds for suNciently small 0 > 0. The proof 
of the lower bound is colnpleted. 
So far, we have proved t.hat { jg() in C([0, 1],E) satisfies an MDP with speed 
{  } and the rate function À defined by (1.2). It is ey to check that 
(c([0. 
1], E), 
I1"11) is a closed subset of (D([0, 1] E), d.), and therefore { g) / in (D([0, 1], E), 
, () J 
also satisfies the MDP with speed 
d) 
of 
(1.) 
(see Dembo and Zeitouni [18, Lemlna 4.1.5 (a)]. The exponential equivalence of 
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{ g'(')" and { s(.)} in (D([0, ll,E),doo), established in Lenuna 2.2, implies now 
b() J  
that - -, ,s"(" in D([0, 1], E)satisfies the MDP with speed {bn) } and the rate 
function A of (1.2) (see Delnbo and Zeitoulfi [18, Theol'em 4.2.13]). Theorem 1.2 is 
proved. 

3. FUNCTIONAL LILs 

Throughout this section, let p(n) = x/2n loglogn, n _> 3. Based on Theoreln 
1.2 and Renmrk 1.2, we can conclude the Functional Laws of the Iterated Loga- 
rithln for the piecewise constant and piecewise lilmar functions, {S,(.); n _> 1} and 
{ç,(.);n _> 1}, respectively. 

Theorem 3.1. Suppose that (1.4) and the following co'ndition 
(n) 
hold. Then with probability 1, the following sequence 

S(.) 

is relatively compact in D([0, 1], E). and the set of its limit points, L(w), is precisely 
the compact set 
K= {f e D([0, 1],E); 2X(f)_< 1}. 
The saine result holds for f '() } 
. [3(n) n>_l 
Pro@ It can be proved by the standard argulnents, see, for exalnple, the proof of 
Deuschel and Stroock [20, Theoreln 1.4.1] or Delnbo and Zajic [17, Çorollary 1]. 

4. CONCLUDING REMARKS 
We have viewed the trajectory probleln as consisting of two major issues. First, 
what is the MDP for partial sums? For this we found good results in the literature, 
see, for instance, Chen [13, 15], Ledoux [26]. Second, how to pass a result from 
the partial sure to the whole trajectory, now that the result holds for partial sums? 
The latter issue is treated careflllly in Section 2. 
We have traced all the proofs and seen that, once the MDP for partial sums 
is assumed, the original assulnption (1.4) is rarely quoted in settling the second 
issue. This, among other things, suggests that any partial sure result may well 
remain true for the corresponding trajectory process. Let us illustrate use of such 
an idea by extending a partial sure result to the trajectory setting, the results in 
Proposition 4.1 and Proposition 4.2 below. In the proof, we will list ail occasions 
of quoting the original assumption, (1.4). It should be pointed out that (1.5) is a 
necessary condition for (1.6) to hold. 
Proposition 4.1. Let b(n) = n  (1/2 < p < 1). Suppose that (1.5) and 
Eexp (/3lIXl] ') < oc for some 2- 1/p < et < 1 and some  = 13(et) > 0 
hold. Then Theorem 1.1 and Theorem 1.2 remain true. 
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MDP with speed 
Again by tracing the proofs of Chen [13, Theorem 2]; [15, Theorem 1], we can 
show that Lemma 2.3 is still truc under the new asSUlnption instead of (1.4). 
Excluding Lemma 2.3, use w ruade of (1.4) only in the proofs of Lemma 2.2 
and (2.12) via the following estimate 
P {l[Xll > (,,)6}  exp(-fl&@))Eexp(fll[Xlll ) 
for  > 0, where  is  in (1.4). The inequality above CllSllrçS [hç desired estimate 
(4.1) lin, s,,p  log(,,P {lIXlll 
u-kl 
for each  > 0. Therefore, ail that remains is to show that (4.1) is valid under 
the new assumption, weaker than (1.4). lndeed, for each  > 0. bv Chebyhev s 
inequality, 
which implies (4.1). Proof is comph'ted. 
Proposition 4.2. Let b(n) = 2n loglogn,  2 3. Suppose that (1.5) and 
exp(lIx1H ) <  for some 0 <  < 1 ad some  = () > 0 
hold. Then Theorem 1.1. Theorem 1. and Theorem 8.1 hold. 
Pro@ Its proof is similar to that of Proposition 4.1. 
Remark 4.1. Proposition 4.2 has improved Theorem 3.1 by weakening the expo- 
nential integrability sumption, (1.4). 

APPENDIX A 
Let (2(. d) be a Polish space and 3; denote thc Polish space of continuous functions 
from [0, 1] to 2( equipped with the metric d(f,g) = sup0<t<l d(f(t),g(t)). Let 
{a(n)} be a positive sequence satisfying 
A sequence of probability measures {tt,; n _> 1} on 3; is said to be exponentially 
tight with speed {a(n)} if for every L > 0. thcre exists a compact set IçL C Y such 
that 
(A.1) limsupa(n) logt, n {tçï} <_ -L, 
where Kï means the complement of 
Theorem A. A sequence of probability measures {n; n > 1} on 3; is exponentially 
tight with speed {a(n)} if; 
(i) For each rational t C [0.1], the sequence {t*,z(t)'n _> 1} of laws induced by 
the projection f(.) ---+ f(t)  Y --, 2( s exponentially ti9ht in (2(, d). that is. for each 
c > O, there exists a compact set L C 2( such that 
limsup a(n)logt*, {f(t) e L} <_ -ct. 
(ii) For all p > O, 
lira lira sup a(n)log p,., { {f; wl(a) 
6--- 0 ---c 
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where for each f E  and all 5>0. 
wj(5)= sup d(f(t),f(s)) 
denotes the modulus of continuity of f. 
Proof. This is essentially the Lemma A.2 of Dclnbo and Zajic [16]. 

APPENDIX B 

Let  be defined by (1.2). The purpose of this section is te show that À, defined 
on (D([O, 1], E), d(., .) ), lins compact level set.s. Since (c([0,1],E),ll- II)as a 
closed subspace of (D([0, 1], E), d(., .)), it sufl=ices for us te show that A restricted 
te (([0,1], E), Il-II) ha ompt evel et, s. T|lrollgllollt, this section, we denote 
by ) the Lebesque measure on [0,1]. Note that A has compact level sers under the 
integrability condition (1.4). 

Lemma B.1. Let Iç.= {ç E C([0, 1];E);À(ç) _<a} fora >0, andlet [x= {gE 
L([O. 1],E);ç(t) = fô g(s)ds for t E [0.1],ç E h.}. Then I is II " llH-uniformly 
integrable, that is to say 

f 
(B.1) lira sup / IIlldA:0. 

Pro@ Given g E/', for each k > O, it follows from the Cauchy-Schwartz inequality 
that 
(B.2) IIg[IHd< X@)< a, 
 IIll_>} 
where q(t) : Jô g(s)ds for t E [0, 1]. (13.1) follows frein (13.2). The lemma is proved. 

Lemma B.2 (Lower semicontinuity of À). /f IIç - 11 
liminf À(ç) > 

--0 as  --, then 

Pro@ It. sutïfices to consider lim inf,_ (ç) = b < e« By passing to a subse- 
quence, we nmy and will assume that liln X(çn) = b, and (ç) <_ b + 1 for all n. 
Let. 9 E LI([0, 1]. E) such that 9(t) E H and ç,(t) = 9(s)ds for t E [0, 1]. 
We shall show that ç is 11" IIH-absohltely contilmous: that is, for every e > 0, 
there exists 3 = 3(e) > 0 such that n E N. 0 _< si < ri _< s2 < 2 _<  "" <_ sn < tn _< 
1, (ti -- si) < 5 imply 

(B.3) 

i=1 

Te prove (B.3). Given e > 0. by Lemma B.1, there exists 5 = 5(e) > 0 such 
that for all n, [., IIg[IHdA < e whenever A(A) < 5. In particular, if s < t and 
}2 It - sl < 5, then 
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Taking account of (B.4), the lower semicontimfity of A and IIç, - çll --' 0, we can 
obtain 
(B.5) e  liLoef (2A(ç(t,) - çn(si))) 1/2  (2A(ç(t,) - ç(si))) 1/2. 
Note that ç(0) = lira ç(0) = 0 6 H. (B.5), together with the definition of A, 
implies that ç(s), ç(t) G H. In return, we have ç(t) G H for ail t G [0, 1] and 
(B.6)  IIç(t) - ç()llH   
which means that ç is II-IIn-absolutely continuous. Note that (B.6) yields also 
that IIç(*)llHd < . 
We next show that there exists g e L([0.1]. E) such that g(t) 6 H and ç(t) = 
g(s)d.s for this define where B is the Borel 
[0,1]. 
To 
([0,1],, ), 
O11 
«-algebra of [0.1], the H-valued martingalc (h, U), where 
 [() (-)] 
/n = 2n ç --ç  1[(-1)/2"j/2") 
3=1 
Because ç is I{" ]l--absolutely continuous, it is of ]]-{{.-bounded variation; that 
is, therc exists a positive constant M < oe, such that if  Ç N and 0  t0 < t < 
 "" < tn  1, then 
(.7)  IIç(t) - ç(t-)ll.  t. 

Since for  Ç N, 

3_-- 1 H 
it follows from (B.7) that sup EI[h,]IH < . By the well-known martingale con- 
vergence theorem, there exists 9 Ç L([ 0. Il,E) satis[ving 9(t) Ç H for t Ç [0, 1] and 
 [[g(s)[[Hds < oe []hn - g[[H = a.e. Next, we that 
such 
that 
lira 
0 
shall 
show 

g(s)ds for t Ç [0, 1]. First of ail, we shall show that 
lira E[[hn - g[[/ = 0. 

To prove (B.8), it. is enough to show {h,} is Il" IIH-uniformly integrable, that is 
(B.9) lim sup / IIhllHdA = 0. 
Toshowthis, let/j=()-(J@nl),j =l,''',2n,then 
2 n 
c = ' sup  II/jll,, <  
n j=l 
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and 

(B.10) 

P 
j=l 

Therefore 

0 t 
ç(t) = a(s)ds. 

Finally, we shall show that b _> À(y)). Let [I  0 = t0 < ri < "'" < tN = 1 be 
a partition of [0,1], where IlIIII /x 
= max Iti - ti-l[ will be taken to be sufficientlv 
I<i<N 
small. Note that (ç) =  A(g)dA. By Jensen's inequality, 
i=1 
A(g)dA > ( -t_I)A 1 gd . 

where 

and gn(1)= ç(1). 

N 
Ï(gn) __ Z(,-i_l)A (gn(i)-gn(i-1)) . 
i=1 ri -- ri-1 
Using the lower senficontinuity of A, we obtain 
N 
(B.13) b=linlinfï(n) oE -(ti-ti_l)A ((ti)-(ti-1) ) 
n--cc i=1 i i--1 
for any partition H, where çn is defined  follows 
ç.(t) = .()d, t e [0, 11, 

gn(t) = y)(ti) 
ri - ri-1 

for t  [ri- 1, il) 

However 
(B.11) [ I[hllHd =  II/jllH. 
IIh, Il  >pi { J:' II/X Il  _>p} 
Now, (B.9) follows ri'oto (B.10), (B.11) and the II-I[H -absolute continuity of ç, 
(B.3). Cnsequently, by the dcfinition of Bo«hner integral, for 0  j < k  2 a, 
, , () 
(B.12) g(s)ds = lira hi(s)ds = ç - ç . 
J3/2n loe Jj/2n 
ecall that the Bo«hner integral h(u) = g(s)ds  [0, 1]  E is COlltinllOllS (see 
Diestel and Uhl [21, Theorem II.2.4]), and therefore, by (B.12) and the contimfity 
ofç, for0Ns<t l, wehave 
f g(s)d = ç(t) - 
[11 particular, for t  [0, 1], we have 
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Let I-I,= be a partition such that III-I11 < ±. Define 
ri i--1 
,d v,=(1} (1), wh ,  0  < 4  < < ,{ = 1. Sin 
 @()lld < , im @. - V]] = 0 a.. urthme, im @. - ] = 
0 a.e. Consequently, taking accotait of the lower semicontinuity of A. we bave 
i,2,On a(V,)  a{) a.., whi«,  wl as »'at' ,ma, impi 
(B.14) ]illiinf A(gn)dA  lilninf A(gn)dA  A(g)dX = Â(). 
Froln (B.13) and (B.14) it follows that 
b k lilninfX(gn ) k 
whi«h provcs the (lesired rcsults. Thc lenmm is proved. 
La n.3. F .«. > 0. hç = {ç- ()  ,} i «,«t i (CI0, q, ]]-]). 
Proof. Note first that: if A C Ç([0.1-E) is such that 
(i) There cxist a compact set L" C E such that (t)  K for ail t  [0.1], ç  A: 
(ii) lira supw(5) = 0, wherc w(5) = sup Iii(t) -(s)]], then , is compact 
in (C([0.1}:E), Il. ]]) (see also de Acosta [3, p. 88]). 
Given  e Ç, let 9 e ([0.11 N) such that (¢) = 9(s)ds for  e [o. 1], then 
Ils(t) - ç(Æ)ll = g()&  IIg()l,&, 
which, together with Lcmma B.1, yidds (ii) for A = 
Civen ç e K« aain, let  e ([0.1], E) such that (0 = (s)dsfort[O, 1]. 
For any t  (0.1], by Jensen's ilmquality 
A((t)) = t2A g(s)ds  t A(g(s))ds 
This provcs (i) for A = L',, wth h" = {x e E;A(x) 
a} is compact in (E, }{. }). Now the compactncss of L follows from the above 
arguments and thc lower scmicontinuity of A (Lemma B.2). Thc proof is completS. 
From Lcmma B.2 and Lcmma B.3, wc can obtain following thcorcm. 
Theorem B.1. Let  be deflned by (1.2). Under condition (1.4) , Â is a rate 
Zti  (c([0, ]; E), . I1). 
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WEIERSTRASS FUNCTIONS WITH RANDOM PHASES 

YANICK HEURTEAUX 

ABSTRACT. Consider the function 
fo(x) = E b-nag(bnx + 0,), 
where b > 1, 0 < a < 1, and g is a non-constant 1-periodic Lipschitz func- 
tion. The phases On are chosen independently with respect to the uniform 
probability measure on [0, 1]. We prove that with probability one, we can 
choose a sequence of scales k , 0 such that for every intervai I of length 
]1[ -- k, t|m oscillation of fo satisfies osc(f0, I) >_ ClI[ '. Moreover, the in- 
equality osc(f0, I) OE cIII + is almost surely true at every scale. When b 
is a transcendental number, these results can be improved: the minoration 
osc(f O, [) >_ CIII c is truc for every choice of the phases On and at every scale. 

The function 

1. INTRODUCTION 

w(x) =  --o cos(2"x), 
=0 
where b > 1 and a _< 1, is probably one of the most famous continuous nowhere 
differentiable functions. This function was introduced by Weierstrass He proved 
that w is nowhere differentiable for some of these values b and a. A few years later, 
Hardy gave the proof for every b > 1 and every a <_ 1 (see [5]). 
More generally, one can consider the fi, uction 
(1.1) f(x) =  -°(x) , 
where g is a 1-periodic Lipschitz function, 1 < b < +oc and 0 <  < 1. Such 
a function will be called a Weierstrass function, and it is easy to prove that f is 
of class C  (see, for example, [3]). Note that the regularity property of f is more 
subtle when a -- 1. In that case, f lies in the Zygnmnd class but is often hot 
Lipschitz. 
A famous conjecture states that the Hausdorff dimension of the graph of f is 
equal to 2 - a (this is the biggest value that one can hope for). There are many 
papers that give general support to this conjecture (see, for example, [11], [10], 
[6], [7], [9], [14], [15]). Of course, one cannot expect this conjecture to be true in 
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general. Suppose for instance that g is of the form g(x) = ç(x) - b-aç(bx) with ç 
a smooth flnction. Then f = ç, and the graI)h has dimension 1. Thus, the genuine 
question is whether thc conjecture is truc when g is hot of the above form. In recent 
work with Thierry Bousch, we proved the following result about the oscillations of 
the function f, which supports the conjccture. 
Theorem 1.1 ([2], [1]). Let g be a 1-periodic Lipschitz function, 1 < b < +oc and 
0 < c < 1. Dejïne f using formula (1.1). There are only two possible mutually 
exclusive cases: 
(i) f is Lips«hitz o,,d IIf'll < 
Or 
(ii) 

(.2) 

there exists a constant C > 0 su«h that for every interval I of length 1II <_ 1, 
osc(f, I) --- sup(f) - iïf(f) > Cll a 
I 

C< -b- + , 
- 1 - I - b-ai 

where osc(g) = sup(g) - inf(g) is the global oscillation of the function g. In 
particular, this constant does hot depend on 0. 
When b is an integer, the function fo is 1-periodic. In the other cases we can 
only ensure that fo is almost periodic (as a uniform limit of a sequence of ahnost 

Moreover, the set of functions g such that (ii) is satisfied is a dense open subset of 
lhe space of 1-periodic Lipschitz functiots (equipped u,ith its standard norm). 
The proof of this result is based on the following functional equation: 
(1.3) f(x) -= g(x) + b-a f(bx), 
which is satisfied by the Wcicrstrass function f. Let us also recall that conclusion 
(1.2) is oftcn present in the literature about Weierstrass-type functions. In [14], 
sut-[icient conditions on g are given which ensure that (1.2) is satisfied. Assuming 
some stronger properties on g, Kaplan et al. ([9]) prove a result similar to Theo- 
rem 1.1 about the bchavior of f. Finally, in [7], Hu and Lau prove a conclusion 
slightly different from (1.2) in the case where the \Vcierstrass-Mandclbrot function 
associated with f is hot identically equal to zero. 
It is well known that conclusion (1.2) ensures that the box-counting dimension 
of the graph of f is greater than 2 - c (sec, for example, [3]). On the other hand, 
McMullen shows in [12] that there are self-affine functions of class C a satisfying 
(1.2) but whose graphs have Hausdorff dimension strictly less than 2 -  (see also 
[14]). These functions are hot \Veierstrass-type functions and do not refute the 
conjecture. 
The purpose of this paper is to study the local oscillation behavior of Weierstrass 
functions with phases. Such a function will be defined bv the formula 
+¢ 
(1.4) fo(x) =  b-''g(b'x + O, 0 , 
r..O 
where g is a 1-periodic Lipschitz function, 1 < b < +oc, 0 < ct < 1 and 0 = (0,),>0 
is a sequence of phases satisfying 0,  [0, 1]. It is easy to check that the function fo 
is still H61der continuous on IR with exponent a (sec, for example, [3] or [8]). More 
precisely, one can prove that the H61der constant C of fo satisfies 
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FIGURE 1. Graph of fo(x) with g(x) = dist (x, Z), b = 2.1 and a = 0.5 

periodic functions). More prccisely, we can establish the following property, which 
states that thc almost periodicity property does not depcnd on the sequence of 
phases O. Such a property will be useful in the proof of the nmin theorcm (see 
Lenmm 4.2). 
Proposition 1.2 (Equi almost periodicity). The family of functions (fe)eE[o,ll TM 
satisfies the following property: 
ve > o, t > o; vo  [o, ]r, v  , a  [,  + e); II'r«/o - foll < e 
(7-«fo is defined by 7-«fo(x) = fo(x ÷ a)). 
Proof. This proposition is a consequence of the compactness of the set [0, 1] . Let 
« > 0 and 0 E [0,1] . We know that the flmction fo is ahnost periodic. So, 
we can find ge > 0 such that for all 6 E l, there exists ae  [6,6÷ge) with 
IIT«ofe- fe[l -- «/3. The application 0  [0, 1]  - fo  Cb(l) being continuous 
(when the set Cb(l) of bounded continuous functions is endowed with the nonn 
Il II), we can find 01, .... 0  e [0.1]  such that the set {fo} is covered bv the halls 
B(fo,e/3 ). Let g = nmx(gol,... ,go,). If 6 E Ii and if fo E B(fo,e/3), we obtain 
I[T«0 fo - fo 11 < e. The conclusion follows if we note that ao E [6, 6 + f). [] 
As we can see in Figure 1, the function fo seems to be irregular, and we would 
like to know if a minoration similar to (1.2) is true for Weierstrass functions with 
phases. In the general case, such a minoration seems to be diflâcult to obtain for 
each value of the sequence 0 = (0,)>_0. That is the reason why we propose to 
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considcr the phases 0n as independent random va.riat)les uniformly distributed in 
[0, 1]. In other words, the set of phases f = [0, 1]  is endowed with the probabilitv 
measure  = dz , where dx denotes thc Lebesgue Incasure on [0, 1]. R.ecall that in 
such a context, B. Hunt proved that when 9(x) = cos x, the Hausdorff dimension of 
the graph of the fimction fo is equal to 2 -  with probability one (sec [8]). Using 
a different method, Szulga recently proved a similar result for the ierstrass- 
Mandelbrot process ([16]). 
If we want to writc a functional equation for the flnction fo, we also have to 
introduce the shift operator on 
(1.5) T : (0n)n 0 Ç   (0n+l)n 0 E  - 
Then, the functional equation can be written  follows: 
(1.6) fo(x) = g(x + 0o) + b-fro(bx)  
It is clear that the l)robability F is T-invariant. Moreover, the 0-1 law, which is 
satisficd by the independent l)hases, statcs that thc «-fi«ld B = 0 T-(B0) is 
constituted of events with trivial probability (B0 is the «-field of Borel sers on Q). 
Then, it is classical to conclu(le that thc strong mixing property 
VA. B  B0, lira F[A O T -"BI = F[A]F[B] 
is satisficd by thc T-invariant probahility F (for elementary results on ergodic thë- 
ory, sec for example [13] or [19]). In particular, thc probability F is ergodic: invari- 
ant Borel sers ha.ve probability 0 or 1. This ergodicity will be the key point of the 
proof of our main theorems (Theorems 3.1 and 3.2). 
Let us now bricfly describe the main results of this paper. We first observe in 
Section 2 that a conclusion like (i) in Theorcm 1.1 is not possible in the random 
context. More precisely, when 9 is not constant, fo is almost surely not Lipschitz. 
Then, ve tan state in Section 3 and prove in Section 4 minorations for the oscilla- 
tions. More prcciscly, ve prove that with probability Olm, the nlinoration 
(1.7) oc(fo, I) 
is truc for every interval I. as soon  g is hOt constant. Moreover, the stronger 
inequality 
(1.8) o(fo, I) 
is valid when II[ = 6, where (5) is a sequence of scales decreasing to zero. In 
particular, the local H61der index of fo is almost surely everywhere equal to 
In Section 5, we deal with the case where b is a transcendental mtmber. In that 
case, we are able to prove that the minoration (1.8) is truc for every interval I and 
for every choice of phases (0)0. In some sense, this means that randomness is 
already present in the number b. 

2. WHEN g IS NOT CONSTANT, fo IS NOT LIPSCHITZ 
The first step of our investigation is to establish that fo cannot be regular (except 
when 9 is constant). This is the aire of the folloving result. 
Theorem 2.1. Let 9 : ]R -- ]R be a 1-periodic Lipschitz function, 1 < b < +oc and 
0 < a < 1. Define fo by formula (1.4), c0 = b1-o-1, and let 
A = {0 E f ; fo is Lipschitz} and Aco = {0  f ; fo is co-Lipschztz}. 
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lE[Al = o or 1 and lE[Ao] = 0 or 1. 
Moreover, the following are equivalent: 
(i) lE[A] = 1: 
(ii) lE[Aco] = 1; 
(iii) g is a constant function. 
Remark 2.2. If 9 is net constant, Theorem 2.1 ensures that with probability one, 
there exist x =fi 9 and v > 0 such that 
(1 + v)llg'll 
I fo(x) - fo(y)l >_ b'- - 1 Ix - yl 
In fact, the theorem also says that there is no more information once one shows 
that fo is ahnost surely net Lipschitz. This remark will be usehfl when proving 
Theorem 3.1. 

Proof of Theorem 2.1. Observing the flmctional equation (1.6), it is clear that A is 
T-invariant. The crgodicity of the shift T then ensurcs that lE[AI = 0 or IlS[A] = 1. 
More precisely, if Ac = {0 G  ; fo is c-Lipschitz} and ç(c) = bc-l(c + 119']]), it 
is easy te check that 
0  Ac == TO  -4(c)  

In other words, 
(2.1) 

4« C T-(A«()) . 

In particular, if c _> c0, then ç(c) _< c and Ac C T-(Ac). We cannot state that 
A« = T-(Ac), but we tan however conclude that IE[A¢] = 0 or IE[A¢] = 1. This 
well-known result is a consequence of the fact that an ergodic invariant probability 
measure has no wandering set of positive measure. Let us propose an dementary 
direct proof, in order te be self-contained. [] 

Lemma 2.3. Let B C f be a Borel set such that B C T- B. Then 

lE[B] = 0 or IE[BI = 1 . 

Pro@ Let C = fi \ B and D = limsup_ T-C. Observe that D is T-invariant, 
and suppose that lE[BI < 1. Then lE[D] > 0 and, bv ergodicity, lE[D] = 1 (hem, we 
use the fact that lE is a finite measure). It follows that II[B] = lE[B  D]. But, if 
B C T-lB, then B C/D = O. So IE[B] = O. [] 

Proof of (i) => (ii). Suppose that lE[A] = 1 and note that A = Uc_>co A. Se, we 
can find sonm c >_ c0 such that lE[A] > 0. It follows that FIAt] = 1. Using (2.1) 
and the T-invariance of the measure lE, we obtain lE[A()] = 1. Iterating this, we 
get IE[A,()] = 1. Taking the limit when r --, oc, we conclude that lE[Ac.] = 1 (c0 
is the unique fixed point of the contraction ç). [] 

Proof of (ii) = (iii). Let x and y be two rem numbers, and write 

f(x) - f(y) = E b-'C'(g(bx + 0) - g(b'y + 0)) . 
rt-O 
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The random vm'iables b - ( g( b' x + 0, ) - g( b ' g + 0, )) are centered and independent 
(here, we use the fact that g is 1-periodic). So, they are orthogonal in L2(P). If 
p  0, the Pythagorean theoren ensures that 
 [(/(.) -/(v)) ] =  -- [(( + o) - ( + o)) ] 
n=O 
 b-PE [(g(¢.r + 0) - g(¢y + Op)) ] 
Replacing x by b-P.r and y by b-Py and using (ii), we get 
.1 
 (a( + t) - a(. + t))dt 5 b%ô b-« - b-l  
Taking the limit when p  , we obtaiu 
(g(« + t) - g(u + = 
t))dt 
[). 
This inlplies that g(x) = g(y).  

3. ON OSCILLATIONS OF fo 
In this section, we state the lllain results concornillg the local behavior of Weier- 
strass functions with randoln phases. 
Theorem 3.1. Let g : IR --. IR be a 1-periodic non-constant Lipschitz function, 
and let 1 < b < +,:x and 0 < c < 1. Define fo by formula (1.4). There ecists 
a constant C> 0 such that for almost every 0  12, we can choose a sequence of 
scales 5 "N 0 such that, for ever9 interval I of length 
osc(fo, 
It seems curions that we have to restrict the conclusion to some sequence of 
scales. In fact, it is easy to sec that the more precise conclusion 
(3.1) BC > 0 ; for almost every 0  
is false in general. Indeed, the set of 0  12 such that ose(fo, I) > C III  for all 
intervals I of length III < 1 is clearly a closed subset of 12. So, conclusion (3.1) 
would imply the smne property for every 0  12. On the other hand, if b is an 
integer and if g(x) = (:r) - b-(b.r), with  a 1-periodic non-constant Lipschitz 
function, then g is a 1-periodic non-constant Lipschitz function. Moreover, fo = 
with 0 = (0.0,--.). So we can find SOllie 0  f such that fo is regular. As we will 
sec in section 5, this phenomenon cannot occur when b is a transcendental number. 
In view of this remark, it is reasonable to ask whether the weaker assertion 
(3.2) for almost every 0  
is truc when g is non-constant. In fact, we do not know if (3.2) is truc for every 
b > 1, but we tan prove ail analogue of (3.2), replacing a by a + e. This is the 
purpose of the following theorem. 
Theorem 3,2. Let g : IR - IR be a 1-periodic non-constant Lipschitz functon. 
and let 1 < b < +oc and 0 < a < 1. Define fo by formula (1.4). For almost every 
0  fL the following conclusion is true: 
(3.3) Ve > 0 , C > O; VI with 
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Using Theorem 3.2 and the fact that fo is c-H61der, we obtain the following 
direct consequence about the local bçhavior of Weierstrass flmctions with plmses. 
Corollary 3.3. Suppose that g is hot constant. The function fo is almost surely 
nowhere differentiable. More precisely, for almost every 0 E , we bave 
lira ln(osc(f0, I)) = c for all z   . 
x6I, , II'N,O In(I/I) 
Inparticular, for almost every 0  ff, the HSlder indez of fo s everywhere equal 
to 

4. PROOF OF THEOREMS 3.1 AND 3.2 

Ve begin with the followiug elelnentary leullna. 
Lemma 4.1 (Transfer lemma). Let x, y   satisfy 

Then 

lY0(y)- Y0(x)l > 

(1 + )llg'll 

ly-xl - b 1-a -1 

for some v > O . 

fo(b-lv)- fo(b-'x)l 

Ib-ly - b-lxl -- b l-a - 1 

Proof. The above lemma is ail easy consequence of the flmctional equation (1.6). 
We have 
fo(b-y) - fo(b-'x) = g(b-v + 0o) - g(b-x + 0o) + b -a (fTo(V) -- fTO(X)) . 
Suppose that x, y and v satisfy the hypothesis of the lemlna. Then 
Ifs(b-lu) - f(b-)l > 1- Iï(U) - f()l _ I(b-U + 00) - (b-lx + 00)1 

Ib-lg-b-lxl 

Ib-ly-b-lxl 

> b,_ a (1 + )llg'll _ IIg'll 
- b -a - 1 
(1 + vbl-)llg'll 
b 1- - 1 

When we apply Lemma 4.1, we shrink the scale by a factor 1lb and obtain 
a stronger estimate (the constant v is replaced by the bigger one vbl-a). Using 
iterations of Lemma 4.1, it is possible, from an estimate of the oscillation of an 
iterate fTo at a big scale, to get an estimate of the oscillation of fo at a small 
scale. Of course, this final minoration will make sense if we are able to control the 
distance lY - xl when we begin to use Lemma 4.1. The following lemma is a step 
in this direction. 

Lemma 4.2. Let v > O. If h,  > O, denote by Mh,t the set of 0  t2 such that for 
every closed interval I of length III = , there exists x  I with x + h  I and 
(1 -I- )llg'll 
Ifo(x ÷ h) - fo(x)l > ---d -  Ihl. 
Then there exist ho > 0 and go > 1 such that 

Y [Mo,to] > o. 
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Pro@ Let. v > 0 and sui)pose that gis not eonsta.nt (this is the only interesting 
case). Aecording to Theoreln 2.1. the set 
Al = {0 e l; Sxe N, Sh > 0; [f0(c+ h)- fo(x)[ >_ (1 +b -3°)[[g'[[- 1 [h{} 
is such that [I] = 1. X can then find ho > 0 sueh that the set 5Iho of 0 satisfying 
(1 + a")llo'll Ihl for some x0 e N 
b t-a - 1 

(4.1) I/o(«o + h,,) - :o(.o)l >_ 
has i)ositive 1)robability. I,et 

b - - 1 
According to Proposition 1.2, we tan find a real mmfl)er f0 > (1 (which can be 
SUl)posed great(,r than 1 and gl'eatcr than 2h0) such that 
VO e , V e ,  e [& + /2); II.fo - foll 5 e 
Let I = [z,z + (o] be a cioscd inl('rval of lcngth go- Supl)ose that 0 satisfies (4.1) 
and ta.ke  = z - z0. XY can fin(l a G [6. 6 + (0/2) such that 

If 'o = a + Xo, we have 

II-ofo - foll < e 

z <_ 7ro <.'o + ho <_ z + (o  

Ik Iorcov(_'r, 
{ Ifo(:o) - fo(xo)l = Ir.fo(:ro) - fo(.ro)l < 
Ifo(:co + ho) - fo(xo + ho)l = Irofo(xo + ho) - fo(xo + h0)l < 
and it follows that 
Ifo(Co + ho) - fo(Co)l > Ifo(.ro + ho) - fo(xo)l - 
Ihol - Ihol - b'-- 
We ha'ce just provcd that 3& o C AIo,t o, and we tan conçlude that 
 can now finish the proof of Theor{,m 3.1. Sut}pose that gis hot constant and 
let v > 0. According to Lemma 4.2, choose h0 > 0 and f0 k 1 such that 
 [M.e.] > o. 
Using the ergodicity of the probal)ility measule D, we ca.n clailn that for ahnost 
every 0  , T(O)  5Iho,e o infinitely ofton (the set limsup,oe T-(lho,eo) is 
T-invariant with positive probability). 
Denote bv E this set of full measure. If 0 G E, we can construct a sequence of 
integers (n)kk0 increasing to +. such that 
T (O)Mo,e o forallkk0. 
Let 0 = gob -. If I is an interval of length I = 0 and if  = b -I. we can find 
a']suchthat.r+ho and 
(1 +v)ll'll 
I,,,.o(« +ho)- fr,,o()l  -- 1 Ihol. 
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Take 

c = llg'll 1ol 
ô(b -- - )  
This constant does not depend on 0 and k and is such that 
osc(fo 1) > fo(b-'k(x + ho))- fo(b-'z)] > t'b"(1-)llg'll b-nh0]-- ClI  
' - - b - - 1 
This completes the proof of Theorem 3.1. [] 
Let us now prove Theorem 3.2. As previously, we choose v > 0, ho > 0 and 
(o >_ 1 such that 
 [Mho,,o] > 0. 
We need more quantitative information al)out the sequence of iterates T n satisfying 
T'(O)  Mho,eo. This witt be given l W the ergodic theorem, whiçh gives us the 
asymptotic behavior of the frequeucy of the returns in .alho,eo (for a simple proof 
of the ergodic theorem, see for example [4], page 98). More precisely, this theorem 
st.ates that ahnost surely, 
1 E r-I T3 p [Mho,eo] 
 j=O ]l/la o,e ° 0 --÷ . 

Ill other Wol'ds, 
(4.2) 

liln n/(O) = 1 almost surely, 
--,¢,, 1« P [Mho,tol 
where (0) is the k-th return in [ho,to. 
Denote by Ê the set of 0 satis'ing (4.2), and let  > 0. If 0  Ê, the sequence 
nk(0)/na_(0) tends to 1 at infinity. So, we can find k0  1 such that 
(4.3) Vk oE ko. n(O) < a + e 
._(0) - 
For simplicity, write  instead of n(0) and let I be an interval of length [Il  
ob-o -. There exists k k ko such that 
«oh -  III  «oh 
We can t.hen choose an intervat .1 C I with tength JI = rob -. Using Lelnma 4.1 
 in the proof of Theorem 3.1. we get 
osc(A,I) > osc(A,J) > (1 +,,b(-})ll'll b_h01 > vlh0l llg'll ib_ I 
- - b l-a-1 - b -a-1 " 
The choice of k0 ensures that 
1 
-   III 
The conclusion follows for every intervat 1 of length III  F0b -°-a if we put 
z, lb01 
C= 
Finally. replacing C by rein(C, C(g0b-o-)+«), we bave the COlWtusion for any 
intervat 1 of tength III  1. 
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Final remark on Theorem 3.2. With the method used in the proof of Theorem 3.2. 
we would bave to know that nk+l(0) -nk(O) is alnlost surely bounded in order 
to conclude that (3.3) is true when e = 0. Of course, this assertion is false: the 
number of iterates between two consecutive returns in lllho,«o cannot be uniformly 
bounded, and the ergodic theorem only ensures that n+(O)/k(O) tends to 1 with 
probability one. 

5. THE CASE WHERE b IS TRANSCENDENTAL 
In the case where b is a transcendental number, ve can improve the conclusion 
of Theorems 3.1 and 3.2. This is the purpose of the fi)llowing result. 
Theorem 5.1. Let b > 1 be a transcendental namber, 0 < o < 1, and g : 
a 1-periodic non-constant Lipschitz fanction. Defi,e fo by formula (1.4). There 
exists a constant C > 0 such that for every 0   ad for every znteal I of 
length I]  1. 
o(f0, I) 
Remark 5.2. In particular, the conclusion is true when 0 = (0, 0,-.-). This seems 
to bc in contradiction with the possible case (i) in Theorem 1.1. In fact, it means 
that if f is ahnost pcriodic, Lipschitz and uon-constant, and if b is a transcendental 
number, thon g(x) = f(x) - b-f(b.r) ('annot i)e 1-periodic. An elementary proof 
of this propcrty can be obtained by Fourier analvsis. For algebraic numbers, such 
a construction is possible. For example, if b =  and if g(x) = cos(2x) - 
2 -a cos(4ux), we obtain 
+ 
f(x) =  b-"g(b") 
which is a regular function. 
Proof of Theorem 5.1. The proof of Theorem 5.1 is a consequence of the following 
lemma, which replaces Lemnm 4.2. 
Lemma 5.3. In the notation of Lemma ., suppose that b is a transcendental 
number. Then, for all v > 0. there exist ho > 0 and êo > 0 su.ch that 
.lh,e =  . 
Proof. Remember the notation in the proof of Lemma 4.2, and let h0 be a positive 
number such that P[k&] > 0. Let t ¢ 0 and define the translation S : Q   by 
(SO), = 0, + rb " (mod 1) . 
 know that in the compact group (/Z)  (equipped with the Haar mesure), 
the translation by an element A = (A,),0 is ergodic if and only if the real 
numbers 1, A0,.-. , A,.-- are Q-independent (it suffices to note that invariant L 2 
functions f are exactly t.hose whose Fourier coefficients satisçv f(n0,---,») = 
e2i(°a°++'a)](no, ... , np) for every p > 0). When b is a transcendental num- 
ber, we can then conclude that S is P-invariant and ergodic. On the other hand. 
as an ey consequence of the relation fso(X - t) = fo(x), we note that k& is 
invariant under the translation S. It follows that P[I&] = 1. If g0 is constructed 
as in Lemma 4.2, we get P[k&,e] = 1. In particular, k&o.e is dense in . More- 
over, if a sequence (0),0 of points of ;&,eo converges to 0  . we knmv that 
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the sequenee fo converges unifornlly to fo. Using the compactness of the intervals 
I = [z, z + 0] of length 0, it is then easy to conclude that 0  kIho,to. Finally, 
Mho,to is dense and closed in . This means that Mho,to = 
Remark 5.4. Let B = (b")0 with b a transcendcntal immber. For every 0 
the projection of the curve (0 + tB)tea in the infinite-dimensional torus (N/Z) n is 
dense in (N/Z) n. If we tan prove that Iho ¢ 0 for some h0 > 0. we tan conclude 
that Mho is dense in  and then obtain anothcr proof of Lemma 5.3. In fact, it is 
ey to construct 0   and ri0 > 0 such that 0  Mho. Let x0  [0, 1] be a point 
where g is minimal, and let Y0 > 0 be such that g(yo +xo) > g(xo). Let 0 = (Xo, x0, 
 "- ,x0,--'). g have 
fo(b-Pyo) - fo(O) =  b-[g(b-Pyo + xo) - g(x0)] 
 b-P[g(yo + xo) - g(x0)] 
> (+3)1'[ 
]b-Pyol if p is suNciently large, 
-- b 1-a -- 1 
and we can conclude that 0  Mb-o. 
Let us now sketch the end of the proof of Thcorem 5.1. The nmin argument is 
the one used for the previous proofs. Let 0  t, and let I be an interval such that 
f0 b-  [  f0b -+. An iteration of Lemnm 5.3 gives 
ose(f0 I) > ose(fo,.) > 
' - - b t--I - b --1 [ç0bJ ' 
where J C I has length f0b -.  can choose C - vlholllg'll 
-- 

6. ON FRACTAL DIMENSION OF THE GRAPH OF fo 
Theorenls 3.2 and 5.1 allow us to estimate the box-counting dimension and the 
packing dimension of the graph of fo. 
Theorem 6.1. Let g : IR -, IR be a 1-periodic non-constant Lipschitz function, 
and let 1 < b < +oc and 0 < a < 1. Define fo by formula (1.4) and F0(A) by 
r0(A) = {(x, f0(«)), x  A}. 
We bave 
(i) for every non-trivial compact interval I. A(F0(I)) = 2 - a almost surely 
a?ld 
(ii) for «very non-trivial interval I, Dira (Fo((/)) = 2 - o almost surely. 
where A(E) and Dira (E) are respectively the box-counting dimension and the pack- 
ing dimension of a set E. 
Moreover, when b is a transcendental number, conclusions (i) and (ii) are valid 
for every 0  f. 
Statement (i) is well known if we have a minoration of the oscillations (see for 
example, [3]). The minoration of the packing dimension is probably less known (for 
general information about the packing dimension, see [18] or [17]). We can state 
the following elementary proposition. 
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Proposition 6.2. Let O < "y < 1. and let h     be a functzon «uch that 
osc(h,I) _ CII] 7 for ali zntervais I of length ]I]  1 . 
Tben. for every interval I witb noem, ptg interior, 
Dira (r(I))  2- 7, 
Proof. ç can sut)pose that I = la. bi is a compaçt interval with a ¢ b (the quantitv 
Dira is c-stable). Remember tha.t if E C , then 
Dira (E) = inf sup(Ei); E Ç Ei whcre the Ei are closed sets . 
i=I 
Suppose that ç([e, bi) C U E.  tan then find a sequence (F) of closed subsets 
of la, bi with 
la,bi= Fi and Vi 1. F(Fi) cEi. 
i=1 
Baire's l)roperty ensures that we can find i0 such that the interior of Fio is hot 
empty. It follows that (Eo) 2 (I'(Fi))  2- % and we can conclude that 
,,,(ç([,b]))  2-. 
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ABSTRACT. Let S be a given set of positive rational primes. Assume that the 
value of the Dedekind zeta function ÇK of a number field /£ is less than or 
equal to zero at some real point j3 in the range  < j3 < 1. We give explicit 
lower bounds on the residue af s ----- 1 of this Dedekind zeta fuuction which 
depend on j3, the absolute value da of the discriminaut of K and the behavior 
in /£ of the rational primes p E S. Now, let k be a rcal abelian numbcr field 
and let j3 be an)" real zero of the zeta function of k. V'e give an upper bound on 
the residue at s = 1 of Çk which depcuds ou j3, dk and the bchavior in k of the 
rational primes p E S. Bv combining these two results, we obtain lower bounds 
for the relative class numbers of some normal CM-fields K which depend on 
the behavior in K of the rational primes p  S. Ve will then show that these 
new lower bounds for relative class numbers are of paramount importance for 
solving, for example, the exponent-two class group problem for the non-normal 
quartic CM-fields. Finally, we will prove Brauer-Siegel-like results about the 
asymptotic behavior of relative class numbers of CM-fields. 

The main results arrived at in this paper are Theorelns 1. 14.22 and 26. 

l. LOWER BOUNDS FOR RESIDUES OF ZETA FUNCTIONS 
Let c > 0 be given (to be selected bclow). It bas long I)een known that Hecke's 
integral representations of Dedekind zeta flnctions ÇK of lnllnber fields / can be 
used to obtain lower bounds for their residues ,- at s -- 1 of the type 
1 - (c/logdi-) _< 3 < 1 and ÇK(3) _< 0 imply nK >_ (1 -- 3)d-)/2(1 + o(1)), 
where o(1) is an error terre that approaches zero as d, -- oc. provided that Iç 
ranges over number fields of a given degree (e.g. sec [Lou2, Proposition Al. Sec also 
[Lan, Chapter XVI, Section 2. Leinma 3. p. 323] for a weaker result). Notice that 
the best lower bound one can deduce (for/3 -- 1 - (2/logdK)) is of the type 
2 
Ç/-(1 -(2/logdK)) _< 0 implies K >_ (1 + o(1)). 
e log dE 
The first aire of this paper is to prove Theorem 1 below, which hot only provides 
a nice treatment of this error terre (by simply getting rid of it!) but also allows us 
to obtain lower bounds for these residues which depend on the behavior in Iç of a 
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finite set S of rational plilneS. Let us first set some notation lf K is an algebraic 
mu,ri»er field and Sis a.ny finite set of positive raliona.1 primes, we define 
n,ç(s) := H H ( -(x())-')-'   
(product of Euler's factors of the Dedekind zeta flmction of K) and 
s := H( +v-/2)  k , 
wiIh the convention Hlç() = A¢ = 1. ()ur first rcsult is a %llows: 

Theoreln 1. 
(1) Let m >_ 1 bc a positive int«9er. There e.rists P2, effective su, ch that for any 
finit« set S EE primes a,d an.q totally imaginary number fleld K of degzee 2n  2m 
. 1/2-n 
and foot diseriminazd Pl := Ulç E P2mAS we bave 
() K  ( - )dç-l/nK(S ') 
(2) Let m  1 bc a posilivc inlc9cr. Let S be atç given finite set of pries 
There ezisls p, effective su«h lhat for an9 totall9 imaginar number field K of 
__ A1/2n 
degree 2n > 2m and foot discrimi?talt 
(2) 
(3) II mn rate p2 = P2,0 = 22 and for smaller values of m we maç take 
p and p2,s for S = 0 and S = {2} as 9iven in Table 1" 

Table 1 

2n k 2m = 

2 4 6 8 10 12 oo 
270 J 1 26 '22 21 '2"/r 2 2"/r 2 
2600 50 25 20 27r  27r 2 27r 2 
36000 650 295 222 194 181 2nA{2} = 167 63--- 

(4) Let K be a totally inta9inary number field o.f degree 2n > 2 and root dis- 
criminant ph >_ 322A{} = 2682.208-.-. 4ssume that (-(fl)  0 for some 
  [1 - (2/logdn), 1), Then. 
(3) «-  ( - )d -)/. 
Proof. See Section 2 below.  
 could have stated this result in the more general setting of the hot necessarilv 
totally imaginary nmnber fields. However. we only aire at using it for obtaining 
good lower bounds for relative class numbers of CM-fields. Notice that, contrarv to 
our previous lower bounds given in [Lou2, Proposition AI. our present lower bounds 
(1) and (2) do hot depend on any pesky error factor 
K = max(1 
- (2n/p-), g exp(-2n/p-)). 
2 
which for a given n approaches 1 as dK  , but which approaches 0 as n   as 
N ranges over CM-fields of bounded root discriminants. Moreover, the rem draw of 
these lower bounds (1) and (2) is that the Euler factors HK(S) being always greater 
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than or equal to one, these t)olmds can be considerably better than the ones without 
the factor HK(S) given in [Lou2, ProI)osition AI. For examl)le, if S = {2} and 2 
si)lits COlnpletely in h, then HK(S) = 4'. ' also l'efer the rea(leÏ to [IIof, Lelmna 
4] and [Sta3, Lemma d] where other simi]ar but less satisfa.ctory lower t)omMs for 
-- are proved (in the case that S : 0). 

2. PROOF OF THEOREM 1 

Let K be a totally illmginary mnld)er field (if degree 2n _> 2. Let (-() and d- 
be the the Dedekind zeta fnnction and the absolute value of the discrilninant of 
K, and set A,- = v%,-/(2) 2 = (pK/2rr) , FK(s) = .4;,-r"(s)O,-(s) raid ,\,- = 
Ress=l(FK) = AKnK. Let 

m,() = --  r"()-,L TM 
27ri 

and x > (I) 

be the inverse Mcllin transfi,rm (f F'(s). H«n(-e, H,(.r) > 0 for x > 0. Let SK(X) 
be the inverse M(.llin tansfi)rm f F (.si. For x > 1) we bave 

1 £ FK(z)x-Zdz = E Hn(xN(Z)/AIç) 

(where/ ranges over the nonzero integral ideals of K). Now, by shifting the vertical 
line of integration "R(z) = c > 1 to the left to the vertical line !t(z) = 1 - a < 0, bv 
using the flnctional equation FK(1 -- z) = FK(:) to corne back to the vertical line 
of integration ?R(z) = a and bv noticing that we pick up only two poles, a simple 
pole of residue ),- at z = 1 and a simple pole of residue --tK at z = 0, we obtain 
that Si-(x) satisfies the following functional equation: 

Using this functional equation and the fact that Fh-(s) is the Mellin transform of 
SK(x), we obtain: 

FK(s) = SK (x)a'* -- SK(1/x)x-* + 

= s(--- 1) + Sh (x)(x  + «t-) d_._fx 

and 
s( AK 1) oe 
(where Z ranges over the nonzero integral ideals of tç), which is nothing but the 
Hecke integral representation of çK(s), in another guise (see [Lan, Chapter XIII, 
Section 3, Theorem 3, p. 260]). Let S be a finite set of distinct rational primes. Set 
N={Z; P lN(Z)PeS}and 

çs(s) = (N(:r)) -= 1-[ II(l - (N(7)))-9-I 
ZES pES 
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(hence, @(1) = IIK(S)). Since H,(x) > 0 for x > O, for 1 - c < s < a we have 

vE(s) 

>- s(71) + . 

H,(xN(Z)/A)(x  + x-) dx 

 + dz 
s(- 1) + / )=a  s z- (l-s) 

= (s-1) + (z)= 

r'"(z)Akçs(z) (z _ + 

1 )dz. 
z-(1-s) 

Since 0 < s < 1 and ÇK(S) _< 0 imply FK(s) <_ O, we obtain: 

Lemma 2. Fix a > 1. Assume that Çt, (ff) <_ 0 for some B E (0, 1). 

Then, 
- 2-i . (z)=, z--- + z-(1-fl) dz. 

1 
From now on, we assume that 

ïn(/3) = 3(2r)n(-mr'(), 

and 

M(fl) = sup 1 + = 

21ri 2 
sup -- 

where 
J, = sin '/-  (T)dT 
decreases towards zero as n > 1 goes to infinity (use F(s)F(1 - s) = r/sin(rs) 
to obtain IF(l/2 + it)[  = r/cosh(rt), and set cosh(nt) = 1/sinT). Notice that 
Jn+4 - OEJn for n > 1 and that ,la = r/2 and J4 = 1. 
In Lemma 2, we shift the vertical line of integration iii(z) = c > 1 leftwards to 
the vertical line R(z) = 1/2. We pick up only ont residue, at z =/3, and obtain: 
n/« >_ /3(1 - fl)(Fn(fl)A--@(/3) - M(/3)@(ll2)Aï,.1/2In) 

> (1 - fl)d-DIg-ÇS(1)(fn(fl ) 

2fld -)1 @(1/2)(2r)'/i,) 
2-  ¢s(1) 

for 

--(1--fl)d(Kfl-1)/2IIK(-')(fn(fl) -- 2/dd(-'/2J'(27r2Asn/2)(2fl-1) 7r- , K ) ' 

Çs(1) 

- H(1 + (N())-") < (1 + p-1/2)2n = A/ 

and çs(1)= 
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Lemma 3. Let 7 = 0.577215--. denote Euler's constant and set f,(13) := 
/3(2n)n(1-)Fn(/3). In the range 0 < 13 <_ l. it foilows that 
f,(13) _> 1 - (1 - 13)f',(1) : 1 + n(1 - 13)(-), + log(2r) - 1) _> 1. 
Proof. Since fn(13) is positive and log-convex in the range/3 > 0 (use the infinite 
product of the F-flmction), fn is convex in the smne range. [] 

Using Lemma a. noticing that 1/2 < 1 -(2/log di*-) _< 13 < 1 implies/3/(213-1) <_ 
(nlogpK -- 1)/(nlogpK --2) and d( -)/2 < e, and noticing that 1 - (2/logdK) _< 
/ <_ 1--(1/logdt,-)ilnplies d-Z)/2/(1- 13) <_ v/-êlog di* -, we finally obtain: 

Proposition 4. Let K be a totaily ima9inary number fieid of degree 2n >_ 2. and 
assume that d- > e 4. 
(1) Assume that ç-(13) < 0 for some 13  [1 -(2/logdi*-), 1). Then, 
(4) n-_>(1--fl)d-O/EK(S)(1--nl°gph--12cJ"(2ASlogpK_2  x  )n/). 
(2) Assume that (K(fl) 5 0 for some fl G [1-(2/logdK), l--(1/logdK)]. Then, 
/ 

(5) 
where 

and where 

St< = 3' + log(2n) - - - 

nlogph - 1 4x/-êJn 

n nlogp--2 7r 2 

RI*- 

(27r2As 
RI*'= \ -K ) logpK 

decreases with PK in the range PI,- >_ e 2/n, i.e., in the range dK >_ e 4. 

Now, we are in a position to complete the proof of Theorem 1. 
(1) Assume that n _> 6 and p/ > 27r2As _> 2rr 2. We have J, _< J6 = 7r/4, 
(nlogpK- 1)/(n logpK -- 2) _< (61og(2n 2) -- 1)/(61og(2rr ) -- 2), and we obtain 

nlogpl -12eA (2n2As) 
 log PK -- 2 7c 2 \ pi*- 

61og(2n 2) - 1 e 1 
-< 61og(2n 2) - 2 2rr < ' 

and (4) yields (1) with P2 = 2n2- Moreover, if S = 0, then As = 1 and 

SK > 7 + log(2rr) - - - 

6 

and (5) yields (2) with/)12 = 2712- 

61og(2n 2) - 1 ,rê log(2neeAs ) > O, 
61og(2n 2)-2 n 

(2) To deal with the cases n _< 5, we use the following values: J1 = 2.62205---, 
32=7r/2,33=1.19814"'- Ja=landJ5= 1 
, 5J1 = 0.87401---. 
(3) For proving the last assertion of Theorem 1, we use (4) with S = {2}, and 
notice that 1-II*-({2}) _> 1/(1 - 4-'), (2rcAs/pt<) 'd2 _< 4 -', J, _< J2 = 7r/2 and 

2eJ, n. log PK -- 1 < e 21og(2700) - 1 < 1. 
7r 2 n log PK -- 2 - 7r 2 log(2700) - 2 - 
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3. LOWER BOUNDS FOR RELATIVE CLASS NUMBERS 
Recall that a number field lç is called a CM-fieM if K is totally imaginary, hence 
of even degree 2, k 2, and if lç is a quadratic extension of its maximal totally real 
subfield k. In that situation, the degree of h is equal to n, the class number h of 
h divides the class umnber 5K of K and we let . = h-/ denote the so-called 
relative class mmd)er of lç. Recall that 
,;) !- = QK« d- 
where t'  2 and QIç  {1,2} m'e lhe munber of complex roots of unity in K 
and lhe Hasse unit index of K, wherc d and dk are the absolute values of the 
discriminants of the mmlwr fiehls K and k and where - and  are the residues 
et .s = 1 .f lhe Dedekind zeta fimclions of Iç and k (see [Was]). XX finally let 
t l/2n alld Pi: dl/n 
pi, = c 1, = "'k denote the root discrimilaltS of K and k, respectively. 
j1/2 
tI«lwe, Pk  PK and da-/d  «- = pa. 
Notutiot 5. Throughout this paper we adop he following notation: 
«. = ( + t - ). 
(In parti«uhr, c2 = 2(- 1) 2 = 1.ff179--- and ca = 2.) 
7 :Euler's constant= 0.577215;i491 -  , 
t = 2 + 7 - log(4g) = 0.046191417392-  -, 
2 = 2 +7- logg : 1.432485ï791152---, 
a = 2 +7- log(/4) = 2.818780140172-.-. 
For  
3.1. Upper bounds for residues of zeta functions. To obtain lower bounds 
for h-, we will use (6), the lower bounds for a- obtained in Theorem 1 and the 
following upper bouuds for : 
Proposition 6. 
(1) (See [LouS. Theorem 1].) Let k be a mmber fleld 4 degree  > 1. Then 
- (2(? - 1) J [ e Pk) n-1 
elogdk )- 
(7) t, < : vn ç log 
 <ff< 1. then 
Moreover. if Çk() = 0 for some fl it the range  _ 
(«'°"-)" : 1 )( . 
(8) 
(2) Let t" be a real abelian number field of dNree 'n > 1 and conductor f > 1. Let 
X denote the 9roup (of order  ) of primitive Dirichlet characters  of cmtductors 
f  1 associated wilh this abelian number field " (of degree z). Set 

(9) Bi,:= H 
I¢A' 

(10) k _< Bt¢. 

Moreover, zf Çk(/3) = 0 for some/3 in the range ½ _</3 < 1, then 
(11) t% <_ (1 -/3) log fk B,. 
4 
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Pro@ According to tlle COlMUctor-discrinlinalt forlllllltt, we do have 
I (l°gfx+l) < (l__logd/+h.,) 
-- n- 1 

( Il II ) rt-- 1 
< -- log pt: + -- " ',(log pk + l) '-1. 
-- Z/--] Z/--I bI = 

Now, h-t. = [I,#e,\-k L(1. \) = [Ii#xe \-k IL( l, \')l. .,,,,ce, lIsing Lel|lIl|a 7 below, 
we obtain (10) and (ll). 
Lenmm 7. Recall tbat we sel t = 2+7 -log(4) = 11.1146.-- . Let  be a primitive 
evev Diricblet cbaracler of coduclor ri > 1. 
(1) (Sec [Loul].) IVe ave 1 
IL(1,)l  (logf x + K1)/2. 
(2) (See [Lou6. Corollary 7BI for tbe q,mdratw case, a,d [LouS, "F}lPOrPl,, 71 for 
-  l°g 2 fa- 
3.2. On rem zeros of Dedekind zeta functions. 
Lemlna 8. Set c := 2( + 1 - 1) 2 (bence.  < ci < 1 < c2 < c3 = 2). 
(1) Let K be a ztormal CM-field. Set c=c3 = 2 if K is abelian atd c = c2 = 
2(- 1) 2 = 1.07... otberwise. Tben. eitber (i) tbere e.rists ome intagi-nary 
quadnztic sahgcld F of K such tbat ÇF(S) od (-(s) bave a common real zero 
in tbe rage 1 - c/logd-  s < 1. or (ii) Çt.(s) bas a real zero in the range 
1 - c/logdtç  s < l. or (iii) ((s)  0 itt tbe rat9e I - c/logdt  s < 1. 
(2) Let K be a hot ecessaril normal CM-fleld sucb tbat (Ç-/(k)(s) k 0 for 
0 < s < 1. 2 Then, eitber(i) Ç(s) bas a realzero in tbe roge 1-2/logdtç  s < 1. 
or (ii) Ç(s)  0 in tbe rm9e 1 - 2/logd-  s < 1. 
(3) Let N denote tbe normal closure of a CM-fleld K. Tben. eitber (i) there 
ests some bna9iary quadrtic subfield F of K sucb tbat ÇF(S) ad ç-(s) bave 
a common real zero iv tbe range 1 - (q/logdN  s < 1, or (ii) çk(s) bas a 
real zero in tbe range 1 - (cl/logdN)  s < 1. or (iii) Ç-(s)  0 i tbe range 
1-(cl/logdNs< 1. 
(4) Let N be the ormal closure of a umber fleld K of degree m. The. tin 
divid«s dÇ :Q] ad th« aegre« [Z"  Ql of N divid«s m. 
(5) Let F be an imaginary quadralic number field. Then, lhe Dedekind zeIa 
function Ç(s) of F bas o r«al zev in th« range 1 - (6/)  s < 1. 
(6) For any e > 0 there exists an ineffective constant c¢ 
Dedekind zeta functions Ç(s) of the i'maginary quadratic nu-tuber fields F bave no 
real zero in lhe range 1 - (c¢/d F )  s < 1. 
Pro@ Let m _ > 1 be a given positive integer and K be a lmlnber field of discrilnilant 
dh > exp(2( + ] - 1)). Then, its Dedekind zeta fimction 
rem zeros in the range 1 - (c/logdç) N s < 1 (this result is a gelieralisation of 
[Sta3, Lelnlna 3] and its proof is given in [LLO. Lelnlna 15]). 

IWe could choose tç 1 ---- 0, by [Rare, Corollary 1]. 
2For example, K is a dihedral or quaternion CM-field of degree 2n _> 8. 
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(1) The abelian case is easy to deal with by using the factorization of ÇK(s) as a 
product of Dirichlet L-series. Let us now deal with the normal case. Assume that we 
are neither in case (i) nor in case (iii). Since we are hOt in case (iii), there exists s in 
the range 1-c2/logdK _< s < 1 such that ÇK(S) > 0. Since t;t- is positive and since 
Ç-(s) has a simple pole at s = 1, it follows that limtl Ç-(s) = -oc.. Hence, 
has a real zerofl ofodd multiplicity n _> 1 in the range 1--c2/logdK <_ si <_ s < 1. 
However, in this range we bave no _< 2. Hence, n = 1. According to [Sta3, 
Tlworem 3], there exists some quadratic subfield F of K such that for E C h" we 
have (E(fl) = 0 if and only if F C E. In particular, ÇF(fl) = 0. Since we are hot in 
case (i), we obtain that F is reM. Hcnce, F Ç k, which implies Ç{fl) = 0, and we 
are in case (ii). 
(2) Easy. 
(3) Assunle that we are neilher in case (i) nor iii case (iii). Since ve are not in case 
{iii), there exists s iii the range 1-(O/log d,,-) _< s < 1 such that ç,,(sl) > 0. Since 
lim,t (L-(s) = -o% there exists some rem zero fl of (n in the range s _< s < 1. 
Since N/K is normal, ÇA- divides (N (sec [FM, Corollary 2] for a short proof of 
the Aralnata-Braucr Theorem). th,nce, ÇN() = 0 and fl is a simple zero of 
for 1 - (c/logdv) < st </3 < 1. According to [Sta3, Theorem 3], there exists a 
quadratic subfield F C N such that E C N and (;(fl) = 0 if and only if F Ç E. 
In particular, F Ç h (fol" ç,-(/3) = 0) and @(ff) = 0. Since we are not in case (i), 
then F is real. Hence F Ç k and ((fl) = 0 and we are in case (ii). 
(4) Use [Sta3, Lemma 7]. 
(5) See [Bes] (the proof of this result was announced to appear in [Hof, Reference 
2] but it has iii fact never been published yet). 
(6) See [Pinl, Siegel's Theorem II], [Pin2. Theorem 1] and [Sie]. [] 

3.3. Lower bounds for relative class nulnbers. We are now in a position to 
obtain lowcr bounds for relative class numbers. 

Theorem 9. Let Iç be a normal CM-field of degree 2-n > 2 and root discriminant 
PK _> 50. Assume that K contains no imaginary quadratzc subfield, or that the 
Dedekind zeta functions of the imaginary quadratic subfields of Ix bave no real zero 
in the range 1 - (c/logdK) <_ s < 1. 
(1) Set c = c2 = 2(v/- - 1) 2. H'e have 

(12) 
- 2ne/2v (nelogpk) - logds: - 2rivée /2- \nelogptc] ' 

and for each entry 2m in Table 2 below, 2-n >_ 2m and p( >_ P2rn imply h Tç > 1. 
(2) Moreover, assume that k is abelian and set c = ca = 2 if K is abelian, and 
set c = c2 = 2(V'- - 1) 2 otherwise. Then. we bave the better lower bound 

(13) 

e/2(2rc)'B logda: - 2nv,e /2 \r(logpa: + l) 

(recall that we bave set n = 2 + 7 - log(4n) = 0.046---), and for each etry 2m 
in Table 3 below. 2n > 2m and PK > P2, imply hï« > 1. 
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I 2m I 4 6 8 10 20 40 100 200 
Il 
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Table 2 (continued) : c = 2 

[ 2m 4 6 8 10 20 40 100 200 
P2m 31300 27200 22600 19400 12500 9000 7000 6230 5383 

Table 3 : k is abelian and c = c2 

212m I 4 6 8 10 20 40 100200 oe  
J 

Table 3 (continued) : k is abelian and c = 2 

2] 2m ] 4 6 8 10 20 40 100200 oe | 

Proof. Let. us first prove (13). 
cases to consider. 
(1) Assume that Çk bas no real zero in the range 1 - ci log d/ 
ÇK(1 -- (c/logdK)) <_ 0 and using (2) with S = 0, we obtain 
C 
K k ec/21ogdK. 

Using (10) we conclude that 

According to Point 1 of Lemma 8, there are t.wo 
< s < 1. Then 

(14) CK > c 
tk -- ec/2Bk logdK" 
(2) AssuIne that Çk(/3) = 0 for some/36 [1 - c/logdK, 1). Then ÇK(/3) = 0 _< 0 
and using (1) with S = 0, we obtain 

2eC/2  

Using (11) we conclude that 
(15) n/- > 2 
tk -- eC/2 Bk log f k " 

Since dK > dk >_ fk and since c _< 2. the right-hand side of (15) is greater than or 
equal t.o the right-hand side of (14), and (14) is always valid. 
Using (14), (9) and (6), we obtain the first lower bound in (13). To deduce the 
. __ t]l/4 ïÇ/2 
second one, we use log dK 2n log PK, V/-/dk > K = p and Pk _ Ph'. 
To prove (12), we use (7) and (8), instead of (10) and (11). [] 

Remarks 10. According to (4), we could easily improve upon (1). For example, 
we have: let K be a totally imaginary number field of degree 2n > 4 and foot. 
,1/2, > 98. Assume that ÇK(/3) < 0 for some /36 [1 - 
discriminant PK := K -- -- 
(2/logdK), 1). Then, 

(16) 

4(1 _/3)e(/3-1)/2. 
_>g 

The reader can easily check that by following the proof of Theorem 9 and by using 
(3), or (16) with S = 0, we can slightly improve upon [LPP, Proposition 4.2]. 

In the saine way, by using Point 2 of Lemma 8 we obtain: 
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Theorem 11. Let K bca hot ecessarily normal CM-field of degree 2n  2 such 
that (ÇK/Ça)(s)  0 for 0 < s < l. 3 Theu, (12) bold witb c = 2. I particular, for 
¢a«h «ntry 2m i'n Table 2, we bave h . > 1 as soon as 2n OE 2nt and Ph  P2m. 
Moreover, if k is abelian, a then (13) holds with c = 2. In paiculaç for each entry 
2m in Table 3, we have h - > 1 as soo as n  m and p-  P2m. 
Finally, by using (3). (7), (8) and Points 3 and 4 of Lemma 8 we obtain: 
Tlmorem 12. Set c = c = 2( - 1) 2. Let K be a ot necessaly ormal CM-field 
of dc9ree 2n > 2 and foot discriiant pK k 2800, let mN := [N  Q] deote the 
dc9rec of its 'normal closure N and assume tbat Iç contais no ima9mary quadratic 

subficld or tba.t thc real zeros in the roT9e 
zcta fmctios of tbe imaginary quadratic 
Th cn. 

1 - (ci log dN) < s < 1 of the Dedekind 
subfield.s of N are hot zeros of K(S). 

('Q lç U'K V//dk 
(17) h. > . 
- 4nmNe¢/2(Tr« log pK) ' 
When dealing with small class mmfi»cr problems for CM-fields K, one can sume 
that eithcr N coutains uo imagimtry quadratic subfield or that (F(S) < 0 in the 
rauge 0 < s < 1 for ail thc imaginary quadratic subfields F of K. which enables one 
to use Theorems 9 and 12. ludced, the class number of any imagiuary quadratic 
subfield of K divides 4h- (see [Oka]), ail the imagiuary quadratic fields of small 
class numbers are known (e.g. thosc of class numbers dividing 4 were determined in 
[Arn], [Bakl], [Bak2], [MW], [Stal I and [Sta.2l), and o,,e ca,, eily check numeri«ally 
that ÇF(s) < 0, 0 < s < 1, for these few imaginary quadratic fields F. However, in 
order to prove in Section 4 a Brauer-Siegel-like result for relative cls nmnbers of 
CM-fields, we prove: 
Theorem 13. Let >> mean that the constants involved in the considered lowev 
bond depend on « onl. Let K be a ,ot necessavil normal CM-field of de9vee 
2 > 4 and root number p-  2800. Assume lhat Kcontains an imaginary 
quadratic subfield F and that ç() = çK(2) = 0 for some   [-(2/logdK), 1). 

(18) h,- > 
ad (ineffectively) 
(1) 

(d /d )½ - 
K k " 

(-e log ph-) n-l ' 

and for each entry 2m in Table 4 below. 2n >_ 2m and pn >_ p2 intply bi,- > 1. 

Table 4: c = c 

] 2rn 6 8 10 12 20 40 100 200 oc 
p2 5-10 aa 5.107 3-106 5-105 50000 15000 7800 6500 5383 

Accordin9 to (12) and (18). it follows that h. -- oc as [Iç " Q] = 2n -- :x) for 
,.11/2n 
normal CM-fields of root discriminants Ph = , >_ c := 5400. 

3For example, K is a dihedral CM-field of degree 2n > 8 as in [LO]. 
4For example, K is a quaternion or a dihedral octic CM-field as in [Lou3] and [Lou5. Section 
2.3.11 . 
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Pro@ According to Point 4 of Lemma 8, we have 1 -  _> -(dh-/dk)-l/' and 
1 -  » (d-/d) -/ (notice that  = p  pç  (d-/d)/). According to 
(3), we have n-  (1 - )/«, and we obtain 
(20) '-  (dKId) -lIn and s- » (d-Idk) -oIn 
Using (6), (7) and (20), we obtain (18) and (19).  

4. A BRAUER-SIEGEL-LIIxE RESULT ON THE ASYMPTOTIC BEHAVIOR OF 
1RELATIVE CLASS NUMBE[IS OF ClkI-FIELDS 

By using our previous lower bounds for relative class nmnbers of CM-fields (sec 
Theorems 9, 12 and 13), we now prove Brauer-Siegel-like results about the as- 
ymptotic behavior of relative class nulnbers of CM-fields. In [H.I, p. 554] it is 
said that the restriction p,- --, oc precludes one from deducing froln the Brauer- 
Siegel theoren that therc exists solne sufficicntly large constant C > 0 such that 
htç -- oc as [K  Q] = 2n -- ec for norlnal CM-ficlds K of foot discrilnilmnts 
dl /2n 
p, = /,. > (7,. The Brauer-Siegel-like results we will obtail here prove that we 
may choose C = 5400. In [Mur2, Proposition 4.1] it is said that as K ranges over 
the set of CM-tields of degrees 2n < 8 and 2n # 4 we have h/,- + oc effectively. 
The Brauer-Siegel-like results we will obtain here prove that for any given B we 
have h«  oc effectively as K ranges over the set. of CM-fields of degrees 2n < 
The Brauer-Siegel-like results for relative class number of CM-fields we are going 
to prove (and which generalize those we obtained in [Lou4] for ilnaginary abelian 
number fields) are as follows: 
Theorem 14. 
(1) Let Iç range over a sequence of normal CM-fields such that their foot dis- 
criminants pt« tend to infinity (e.g. let Iç range over a sequence of imaginary 
abelian number fields ), or let K range over a sequence of hot necessarily normal 
CM-fields of a given degree. Let o(1) denote an error terre that tends to zero as 
goes to infinity. 
We bave 
(21) (½ + o(1))log(dl,-/dk)>_ log h,- >_ ( + o(1))log(dtt./dk), 
1 
i.e., log h,. is as9mptotic to  log(d,-/dk), which implies that 
(22) h¢ » d- +°(1). 
The upper bound on log hï,. in (21) is effective and explicit. 
If Iç contains no imaginary quadratic subfield, then the lower bouds for log 
in (21) and (22) are effective and explicit. 
If Iç contains an imaginary quadratic subfield, then the lower bounds for 
in (21) and (22) are not effective, but we bave the following effective and explicit 
weaker lower bound: 
(23) log ht. > (1 1 ) 
- - - + o(1) log(dr4/d), 

5For in that case it follows that p/,- OE Vr (see [Murl, Corollary 1]). 
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whi«h in,plies the following effe«twe and explicit lower bound: 
1 
(24) 7,- >> dt- 
Finally. in the situations where the error terres o(1) in (21). (22), (23) and (24) 
are declared fo be effective and explicit, they are of the type o(1) = O((loglogpK)/ 
log PK)- 
(2) If K ranges over hot necessarily normal CM-fields of a 9iven degree, then 
h,  oe effectively and explicitly as dK  oe. 
For any 9iven h >_ 1 there exists Ph effective such that hç > h for all normal 
CM-fields K of roof discriminants PK > Ph. 
In particular, h K > 1 for all normal CM-flelds h of foot discriminants PK > 
Pl = 40000. 
Moreover, hl --, ,2 as [K  Q] = 2n --, oc .for normal CM-fields K of foot 
3/2' > C = 5400. 
discriminants pl = "a - 

4.1. Proof of Theorem 14. 

Lemma 15. Let h be a CM-field of degree 2n. Then, 
1 
(25) l°ghc -< (7 + o(1))log(dn-/d) 
where o(1) = O( (log log pK ) / lOg pK ) is an ezplicit errer terre that tends to zero as 
flK goes to infinity. 
Proof. Since ç(w) k /2 for w k 2 and since ç(w) must divide 2n, we have 
WK  8n . Moreover, dK/dk  dK = pV. Hence, using [LouT, Corollary 3], we 
obtain 
( ) ( ) 
h <2QKwK e log(d/d) <32n2 logpK 
and the desired result, by using log(dK/dk)  log(d(4) = n 1ogpK.  
1. The first point of Theorem 14 follows from Lemnm 15 and Theorems 9. 12 
and 13 (to prove the last assertion of the first point of Theorem 14. recall that 
1/2 
log(dK/d)  log(d K ) = n logp). 
2. The first and second assertions of the second point of Theoreln 14 follow 
from the first point of Theorem 14 (for CM-fields of degrees 2n > 4) and from the 
following known results (for CM-fields of degrees 2n  4): 
Lemma 16. 
(1) (Sec [Oes].) For every e > 0 we bave an effective and explicit lower bound 
h  » log -« dF for the class numbers hF of the imaginary quadratic fields F. 
(2) If K = FF2 is an imaginary bicyclic biquadratic fleld (where F and F2 
denote the two imaginary quadmtic subfields of K), then dK/dk = dF dF and 

h- -- %IK hFlhF . 

Hence, we have an effective and explicit lower bound hïv » log - dK. 
(3) If K is a non-normal quartic CM-field, then its normal closure N is a dihedral 
octic CM-field, dN/dN+ = (dK/d) , and 
QN (h7)2. 
h- 2 
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1 
Therefore, log h . is effectively and explicitly asymptotic to  log(dK /dk ). 
3. The third assertion of the second point of TheoreIn 14 follows from the fact 
that if h- = 1 and Iç contains ail imaginary quadratic field F, then hF divides 4 
(sec [Oka]). Hence F is known (sec [Ami, [Bakl], [Bak2], [MW], [Stal] and [Sta2]), 
and munerical computations easily yield that ÇF(S) < 0 for these few imaginary 
quadratic fields F. Hence, the first point of Theorem 9 yields that h- > 1 for 
ail normal CM-fields K of foot discriminants PE > 40000. (W'e could also use 
Theorem 13 and the solution of the class number one problem for the imaginary 
quadratic fiel& (sec [Bakl] and [St.al]) and for the imaginary biquadratic bicyclic 
fields (sec [BP]), but we would obtain the weaker following result: h. > 1 for ail 
normal CM-fields K of foot discriminants PK > 7  10n.) 
4. Finally, the fourth assertion of the second point of Theorem 14 follows from 
the last assertion of Theorem 13. 

Remarks 17. It is possible to deduce from the usual Brauer-Siegel theorem for 
class nulnbers of lmlnber fields the following Brauer-Siegel-like result for relative 
class nmnbers of norlllal CM-fields, which illlproves llpOll [HH, Lemma 4] (which is 
given only for CM-fields of a given degree) but is less satisfactory than our previous 
Theorem 14 (for it is incffcctive in the case that N contains no imagilmry quadratic 
subfield): 
Theorem 18. If N ranges over a sequence of normal CM-fields such that their 
foot discriminants PN tend to infinity, then we bave 
1 
loghv ,  log(dN/dN+), 

which implies 

]v >> d; +°(1) 
where o(1) is an error terre that tends to zero as PN goes to infinity. 

5. BETTER LOWER BOUNDS FOR RELATIVE CLASS NUMBERS 

The aim of this section is to ilnprove upon, in the case that k is abelian, the 
explicit lower bounds for relative class numbers of CM-fields K that we obtained 
in the previous section. To this end, we choose S = {2} and use Theoreln 1 to get 
better lower bounds (depending on the behavior of 2 in K) for the terre nK in (6). 
Moreover, using the results of [Lou9] we will be able to get better upper bounds 
(depending on the behavior of 2 in k) for the terre nk in (6). Putting everything 
together, we will obtain Theorem 22, which improves upon the lower bounds for 
relative class numbers that we obtained in Theoreln 9. 

5.1. Upper bounds for IL(l, 
Lemma 19. Let X be a primitive even Dirichlet character of conductor fx > 1. 
(1) (See [Loul] and [Lou9].) Set 

(26) 

gl = 2 + 3' - log(47r) = 0.046 -.. if ,((2) = +1, 
x:= =2+3'-logTr=l.432--- if ,((2)=0, 
3=2+3'-log(Tr/4)=2.818-.. if ,((2) 70,+1. 

6We could choose  = 0 and  = log 4 = 1.386-- -, by [Ram, Corollaries 1 and 2]. 
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(27) 
(2) 

(28) 

We have 
1 x(2) i_l(log f 2 +) 
IL(I, \)1-< l 1 
(See [Lou6, Corollary 7BI for the quadratic case and [Lou8, Theorem 7] for 
the general case.) If L(/3, \') = 0 for some /3  [1/2, 1), then 
1 
[L(1, y)[ < log  f. 
- 8 

5.2. Upper bounds for residues of zeta functions. 

Proposition 20. Let k be a real abelian nurnber field of degree n > 1 and conductor 
fa > 1. Let Xk denote the group (of order n) of primitive Dirichlet characters ,\ 
of conductors fx > 1 associated with this abelian number fleld k (of degree n). 
Let e, f and g = ni(et) denote the index of ramification of 2 in k, the inertia 
degree of 2 in k and the number of priTne ideals of k above 2. respectively. ( Hence, 
IIk({2}) = (1 -- 2-f)-g.) Set 
1 
(29) t% :------ Z x (with as in(26)) 

(30) < <,,f,g := 

(g - l)tq + (n - fg)2 + (fg - g)3 

(hence 0 <  <_ m,f,a <_ s <_ 3) and 

(31) B({2}) :=  H (logf\ 

Vu 
+ tx) < (log pt -1- I'k) 
- 2.4,-  

(32) 

 < n({2})({2}). 

Moreover, if Ça(/3) = 0 for some /3  [1/2.1), then 
(33) nk < 3(1 --/3) log ft 
-- 4 II({2})Ba({2}). 

Pro@ To deduce (30) frolll (29), we notice that, according to [B,'as, Theorem 3.7], 
we have #{¥  X; t(2) = 1} = g, #{\  Xt¢; ¥(2) = 0} = n- fg and 
#{\  Xk; \(2) #0, 1} =n-9-(n-fg) = f9-9. Usingthefact that the 
geometric mean is less than or equal to the arithlnetic mean and the conductor- 
discriminant formula I-I#,(ex fx = d = p., we do bave 
1 ( 1 ),-1 v (logpk+)_  
B({2}) < -,_, - Z (logf, + ) = 4'-  . 
-2. n 1 2- 
l#xiX 

Noticing that 

(log f, + te,c), 
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which proves (32). Now, if Çk(/3) : 0, then L(/3, \o) = 0 foi" some 1 -# 4o E Xk. 
Using (28) we obtaili 

1 -/3 3 \'0(2) [_, . log 2 f-,o 
IL( l, \o)[ _< 1 -/3 log 2 f., -< --8 oeil 2 

< 3(1 -/3) logfn l1 k0(2) ]_1 log f o 
- 4 2 ._4. 

which, in using (27) for ail ti,e "t E -\'n \ {1, \0}, yields (33). 

Remarks 21. Notice that in the specia.1 case that the prime 2 is inert in the real 
abelian number field k of degree n, then (31) and (32) yield 
Vn (log p ÷ n31n--1 
mk -< 2n-l(2n - 1) 
whereas (9) and (10) only yield nk _< v (log p. + t«  )n- /2,-  

5.3. Lower bounds for relative class nulnbers. 

Theorem 22. Let K be a normal CM-field of degree 20 >_ 2m > 2 and root 
discriminant PK >_ P2m.{2} with P2m,{2} as in Table l. Assume thal k is abelian. Set 
c = c3 = 2 if K is abelian, and set c = c2 = 2(vf- 1) 2 otherwise. Assume that K 
contains no imaginary quadratic subfield or that the Dedekind zeta functions of the 
imaginary quadratic subfields of K bave no real zero in the range 1 - (ci log dK) _< 
s < 1. Then, 

« QKw,-II,-/({2})v/---/d 
(34) h,- _> e/2 (2n),Bt:({2}) logd/,. 
wth Bi:({2}) as "in (31) and IIK/k({2}) = IIK({2})/IIt:({2}). 
Therefore, settin9 C,,.t,, = 2(1 + 2-I) -/'  [4/3.2), we bave 
(35) h7,-> « ( c"94h )  
- 2ove/2 n(logpK + 
(with f, g and nn,L as in Proposition 20). In particular, for each entry 2n in 
Table 6 below, we bave h . > 1 as soon as Pic >_ P2,. 
Table 6 (compare with Table 3) 
2n I 4 6 8 10 20 40 100 200 
p forc=c2 I 5217 2ï04 lï07 1228 538 310 206 181 
p for c = 2 4233 2344 1530 1124 513 303 205 180 
Pro@ According to Points 1 and 2 of Lemma 8, there are two cases to consider. 
(1) Assumethat Ç hasnorealzerointberange 1--c/logdK _< s < 1. Then 
ç(1 - (c/logd)) <_ 0 and using (2) with S = {2}, we obtain 
th" >-- eC/2 log dK " 

Using (32) we conclude that 

(36) 

__ > dI/({2}) 
k -- ec/2Bk({2}) logda-" 
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(2) Assume that Ça(ff) = 0 for some fl  [1 - (ci log dK), 1). Then ÇK(fl) = 0 _< 0 
and using (1) with S = {2}, we obtain 

a'K > 
Using (33) we conclude that 
(37) /" 

(1 - fl)IIK/k({2}) 

2ec/2 

- ¢c/2({2}) 3c logA" 

Now, if n >_ 3, then d/x  d/  f (see [l[ill'l, Corollary 1]), and if n = 2. then 
K is cvclic, quartic and here again dK = lâ fk _ > f3k. Hence, we always have 
3c 
 log f 5 3 log fk  log dK 
(for c G 2). Therefore, the right-hand side of (37) is greater than or equal to the 
right-hand side of (36), and (36) is always valid. Using (36), (31) and (6), we obtain 
= _ 3/4 = p[2, Pk < 
(34). To deduce (35), we use log dK 2n log p,  > K _ PK 
alld 
(3) "n/({}) = 2(1- /)- k 
(where 2 = -1, 0 or 1 according  the prime ideals of k above 2 are inert. 
ramified or split in the quadratic extension K/k). In particular, [IK/k({2}) = 1 if 
2 is ramified in tç/k. Finally, since ,I, > O, the right-hand side of (35) increes 
with PK k e 2. Hence, for a given n and and a given pa- k 55 > e 4 ve can 
easily compute the minima of the right-hand sides of (35) over all the pairs (f, 9) 
with f k 1 and n k 1 such that rg divides n, and these minima incree with 
PK  55 > e 4. This inakes it easy to compute p2 for any given entry 2n in Table 
6. 

In the saine way, by using Point 2 of Lemma 8 we also obtain: 

Theorem 23. Let K be a hot necessarily normal CM-field of degree 2n >_ 2 suc 
that (ÇK/Çk)(s) _> 0 for 0 < s < 1 and such that k is abelian. 7 Then (34) and (35) 
hold wzth c = 2. In particular, for each entry 2n in Table 6. we bave h.,. > 1 as 
soon as PK _ P2n. 

6. AN APPLICATION OF THESE BETTER LOWER BOUNDS 
The aire of this section is to give an example showing the paramount usefulness 
of Theorem 22 when dealing with class group problems for various types of CM- 
fields for which Theorem 9 is of less or no practical usefulness. In [Lou5] we proved 
that if K is a non-normal quartic CM-field, then 
v/e 
(39) h.ç _ 12(log(dK/dk) + 0.052) 2 
(notice that according to its proof, there is a nfisprint in the statement of the lower 
bound [Lou5, Corollary 15]). Wc will now improvc upon this lowcr bound. 

7For example, /x" is a quaternion or a dihedral octic CM-field as in [Lou3], and [Lou5, Section 
2.3.11 . 
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(42) 
where 

Lemma 24. Let N be the normal closure of a non-normal quartic CM-.field Iç. 
Hence, N is a dihedral octic CM-field. Tben, 
QNIIN/N+({2 }) v/dN/dN+ 
(40) h  _> 
4«7r 4 BN+ ({2})log dN 
(.for PN -- 222), and 
(41) BN+({2})]ogdN <_ (log(dn-/d«) + 3)4/128. 
Pro@ To get (40), use (34) with c = 2 (see Theore]n 23). Let us now prove (41). 
Let L1 = k, L2 and L+ be the three rem quadratic subfields of N +, the extension 
N/L+ being cyclic quartic, md let , 2 and + be the constants x defined in 
(26) sociated with the three quadratic characters k of these three rem quadratic 
fie]ds. It is known that L2 = Q() and that dL divides dç/d. S[nce 
d+ divides the product dd (for N+/Q is biquadratic bicyclic), we conclude 
that dL+ divides dn-/d. Upon using the bomd dN  (dN/dN+)  = (d-/d) 4, we 
obtain (see (31)): 
128BN+ ({ 2}) ]og dN 
 4(lod + )(log(d-/dî.)+ )(log(d./d»)+ +)log(d-/d) 
 (log(d./d) + , + 2)2(log(dn-/d)+ +)log(d/d) 
(for4ab(a+b)  fora0andb0) 
 (log(dn-/d) + (2 + 22 + +)/4) 4 
(for a2bc  ((2a + b + c)/4) 4 for a  0, b  0 and c  0). 
Finally, since either 2 splits in one of the three quadratic subfields of k, or 2 rmnifies 
in at least two of the three quadratic subfields of k, we bave (2 
(4n3 + 1)/4 = 2.830327.--. 
Theorem 25. Let K be a non-normal qnartzc Cçfield. Assume that p- OE 222. 
h  (log(d-/d) + 3)   C(log(dh./d) + 3)  

9v/êTr2/32 = 4.57656-.- if2 is hot ra.rnified 
CK = 3V/êTr2/16 = 3.05104- if 2 is ramified in K, 
vrêr2/8 = 2.03402... g 2 is totally ranified in K. 
Pro@ Let N denote the normal closure of Iç. Then N is a dihedral octic CM-field. 
Since ÇN/ÇN+ = (ÇK/Çk) 2, it follows that dN/dN+ = (dK/dk) 2, Y[N/N+({2}) = 
(IIK/k({2})) 2 and/ = QN(I-)2/2. Using (40) and Lemnm 24, we obtain the first 
lower bound for h. 
As for the second lower bound, we use 
X()k) --1 / 1 if 2 is totally ramified in K, 
IIK/k({2}) = H (1 Nk/Q(,pk))_ _> [2/3 if2is ramified in 
PI(2) (2/3) 2 in all cases, 
where "Pk ranges over the prime ideals of k above the rational prime 2 and \ denotes 
the quadratic character associated with the extension Iç/k. [] 
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Theorem 26. ff the ideal class group of a non-normal quartic CM-field ti is of 
exponent <_ 2. then hï- _ 2 5 and dK/d _< 3- 10 6. 

_  = ,41/4 > (dK/dk) U4 > 
Proof. \Ve assume that dK/dk > 3 109, which implies Pt K -- - 
222. Let t denote the lmlnber of rationa] prilles ralnified in k/Q and ]et T be the 
number of prime ideals ramified in lç/L'. Let Pi = 3 <_ P2 : 3 < P3 = 5 _< P4 = 
5 < P5 = 7-- - be the nondecreasing sequence of ail the odd primes, each one being 
I" 
repeated twice aud set 5,. = I-It-=l P-- In the saine way, set/51 = 3 < lb2 = 3 </53 = 
4 < ifiEE = 4 </35 = 5--- (where for k _> 5 we set/3k = Pk-2) and set ,- = I-If.=1Pk- 
If 2 is hot ramified in K, then dh/dt. > 6t+r, whereas if 2 is ramified in Il, then 
dK/d >_ t+T. Now, assume that thc ideal class group of a lon-norlnal quartic 
CM-ficld N is of expouen < 2. Tllen hï« _< 2 t+T-2 (see [LYK. Corollary 17]). Now 
there are two cases to COllsider. 
First, assulne thal 2 is hot ramified in K. Using the lower bound (42) (which is 
an increasing fimction of dK/d.), we obtain 

V/tt+T with CK = 9V/êr2/32. 
"2t+T-2 k hï a- >_ CK(log(6t+T) ÷ 3) 2 

which ilnplies t + T _< 16. h,- _< 214 and d1/de <_ 1.5- l016, bv using (42). 
Second. assulne that 2 is ranfified in K. Using the lower bound (42) (vhich is 
an increasing fimction of dt,-/dt.), we obtain 

2t+T-2 k h ,- k 

Ct,-(log(t+T) + 3) 2 

with CK = 3v/êTr2/16. 

which implies t + T < 17, h,- _< 215 and dt,-/dn <_ 2.8- 10 6, by using (42). [] 

Remarks 27. 
(1) If we use (13) with c = 2 (see Theorell 11) and Lelnlna 24 we obtain the 
following lower bounds for relative class mmbers of non-normal quartic CM-fields: 

V/dK/dK+ where CK = Vér2/2 = 8.13611--- 
hï« > Ct,-(log(dK/dt,-+ ) + 5/1/4) 2 

Using this lower bound, we would only obtain that if the ideal class group of a 
lmn-norlnal quartic CM-field Il is of exponent _< 2, thell h- _< 2 t6 and dK/dK+ <_ 
9- 10 , a 30-fold less satisfactory bound than the previous one. 
(2) If we had used (39). we would only have obtained that if the idem class 
group of a non-normal quartic CM-field/x" is of exponent _< 2. then h- _< 21 and 
dK/d <_ 1019, a 333-fold less satisfactory bound than the previous one (and in fact 
a bound of no practical use). 
(3) The desire to determine all the non-rionnal quartic CM-fields and ail the 
dihedral octic CM-fields with idem class groups of exponents _< 2 has been a con- 
tinuous incentive to obtain here as good as possible lower bounds for relative class 
numbers of CM-fields. These determinations have now been colnpleted and tan be 
round in [LYK]. 
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PRIMITIVE FREE CUBICS 
WITH SPECIFIED NORM AND TRACE 

SOPHIE HUCZYNSKA AND STEPHEN D. COHEN 

ABSTRACT The existence of a primitive free (normal) cubic x 3 - ax 2 + cx - b 
over a finite field F with arbitrary specified values of a ( 0) and b (primitive) 
is guaranteed. This is the most delicate case of a general existence theorem 
whose proof is thereby completed. 

1. INTRODUCTION 
Given q, a power of a prime p, let F denote thc finite field GF(q) of order q 
and. for a given positive integer n, let E delote its extension GF(q n) of degree . 
A primitive element of E is a generator of the cyclic group E*. The extension E 
is also cyclic when viewed  an FG-module, G being the Galois group of E over 
F, and a generator is called a free element of E over F. The cote result linking 
additive and multiplicative structure  the primitive normal basis theorem  is 
that there exists a  E, sinmltaneously primitive and free over F. Existence of 
such an element for every extension w demonstrated by Lenstra and Schoof [LeSc] 
(completing work by Carlitz ([Ca.1], [Ca2]) and Davenport [Da]). A computer-free 
proof of the primitive normal bis theorem is given in [ÇoHul]. 
It is natural to k whether the result of the Primitive Normal Basis Theorem 
can be extended by imposing additional conditions on the priInitive free element. 
In particular, we may wish to prescribe the norm or trace of a primitive free ele- 
ment, equivalent to speciÇving the constant term or the coecient of x n- of the 
corresponding primitive free polynomial. In [CoHal], it w shown that, given an 
arbitrary nonzero elem«t a  F, there exists a primitive element w of E, free over 
F, such that w h (E, F)-trace a in F, i.e.. TrE/F(W) := i=O wq = a. rther- 
more, in [CoHa2] it was shown that, given an arbitrary primitive element b of F, 
there exists a primitive element w of E, free over F, with (E, F)-norm b in F, i.e., 
n--i i  - 
N/F() := .=0 q =  - = b. 
In [CoHa2], Cohen and Hachenberger posed the following question, known  the 
PFNT problem. (A similar description of the above problems would be as PFT, 
PFN respectively, and later we refer to the analogous PNT problem.) 
Problem 1.1. Given a finite extension E/F of Galois fields, a primitive element 
b in F and a nonzero element a in F, does there exist a primitive element w  E, 
free over F, whose (E, F)-norm and trace equal b and a respectively? Equivalently, 
amongst all polynomials io cixi (ci  F) of degree  over F, does there exist 
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one that is primitive and free, with Ch_ 1 : -a and c0 = (-1)'b'? If so for each pair 
(a, b), then the pair (q, n) corresponding to E/F is called a PFNT pair. 
Observe that the problem is meaningflfl only for , > 3. Clearly the strongest 
results (and correspondingly those most challenging to prove) occur for snmll n 
since the corresponding polynomials have fewest "degrees of freedom". The PFNT 
problem was resolved for all  >_ 5 in [Ci {Theorem 1.1): it was observed that the 
J = I case was delicate while the n = 3 case might prove entirely intractable. The 
J = 4 case was solved in [CoHu2], using a modified version of the ,z > 5 approach. 
In what follows, we solve the PFNT problem in the affirmative for n = 3. Ex- 
pressing the result in terres of polynomials, we show that: for any prime power q, 
given a, b  F* (b primitive), at least one of the q cubic polynomials x a-ax2+cx-b 
(c  F) is primitive and free. Perhaps surprisingly, there are no exceptions. 
We have therefore completed the final stage in solving the general PFNT prob- 
lem. i.e., we have establishcd the existence of a primitive free element with pre- 
scribcd norm and trace for every extension. The result is summarized in the fol- 
lowing thcorem. 
Theorem 1.2. Let q be a prieure power ad n > 3 an integer. Then (q,) is a 
PFNT pair. 
Thc basic technique ([CoHa2]) of expressing the number of elenmnts with the 
dcsircd propertics in terres of Gauss sums over E yiclds, if applied directly, estimates 
in terlns of thc numbcrs of prime factors of qn _ 1 and irreduciblc factors of x n - 1. 
This establishes the result for large t but is ilmdequate when n is sinall. In [Coi, 
use of a sieve on both the additive and Inultiplicative parts produces an expression 
in terins of the numbers of prime (respectivcly, irreducible) factors of divisors of 
qn _ 1 (respcctively, x n - 1), which are estinmted as previously; this approach is 
Inore successful in dealing with Slnall n but relnains inappropriate for r < 5. In 
this paper, we exploit the idiosyncrasies of the situation when n = 3 (allowing us 
to reduce the PFNT problem to the silnpler PNT problem ill some cases) and, 
crucially, employ "exterlml'" results to estfinate appropriate quantities (i.e., we no 
longer depend exelusivelv on the estimates derived from the initial Gauss sure 
formulation). The extreme delicacy of the n = 3 case means that the reduetions and 
improvements which we apply to the basic teclmique are hot Inerely conveniences, 
but. are vital ill establishing the result. Filmlly, a number of vahtes of q <_ 256 (34 
iii ail) had to be checked «olnputalionally. 

2,. P1RELIMINA1RIES 

We begin by making solne reductions to the problem and fornmlating the basic 
theory. The aecount will be as self-contained as possible, but to avoid excessive 
repetition, reference will be Inade to earlier work where appropriate. 
By Proposition 4.1 of [CoHa2], (q, n) is a PFNT pair whenever q - 1 divides n 
(so we inay assulne that q # 2, 4, in the case when n = 3). 
From now on, suppose that a,b  F, with a # 0 and b a prilnitive element, are 
given. 
Let m = m(q.n) be the greatest divisor of q' - 1 that is relatively prime to 
qn-1  In [Coi it was demonstrated that, if 
q- 1 (so in particular m divides (q-1)(n,q-1)" 
w  E bas (E, F)-norm b, then to guarantee that u, is primitive it sufiïces to show 
that w is m-free in E (i.e., that w = v d, where v  E and d[m, implies d = 1). 
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Analogously fol" the additive part, let /1I = BI(q, n) be the monic divisor of 
r ' - 1 (over F) of maxilnal degree that is prilne t.o .r - 1. So M = x'-i where 
n = n0p t, p = charF and p { n0. It was shoxvn in [Coi that, if w  E has (nonzero) 
(E, F)-trace a, then to guarantee that iv is free over F it suffices to show that w is 
M-free in/ (i.e., that w = h°(v), where v  E and h is an F-divisor of M, implies 
h=l). 
Define N(t, T) to be the lmlnber of elelnents of E that 
(i) are t-fl'ee (t  Z, tire ), 
(ii) arc T-froc (T(x)  F[x], TI.r - 1), 
(iii) bave norm b, and 
(iv) have trace a. 
Write zr(t. T) for q(q - 1)N(t, T). Assulne throughout that t ] m and T Ix ' - 1. 
Next, we express the characteristic functions of the four subsets of E (or E*) 
defined by the conditions (i)-(iv) in terres of characters on E or F. 
I. The set of w  E* with NE/F('W) = b. The çharacteristic flmction of the mil»set 
of E* eon|rising elements with llllrlll b is 
I 
q_   .((,)-1), 
vEF* 
where pa. denotes the group of multiplicative charactes of F*, and the norln NEWoe 
is abbreviated to N. 

II. The set of w  E* with TrE/F(W ) = a. The characteristic function of the 
subset of E comprising elelnents with trace a is 
  («(()- )). 
q cF 
where  is the canonicM additive clmracter of F raid the trace Tr/oe is abbreviated 
to T. 

III. The set of w  E* [hat are t-free. The characteristic function for the subset 
of t-free elements (tire) of E* is 
O(t) f oa(w), w 
where O(t) = ¢ t--2) , rd denotes a character of order d (dira) in Ê* and. using the 
notation introduced in [Coi, the integral notation is shorthand for a weighted suln: 
dit (d) (d) 
dit 
IV. The set of w  E that a T-free over F. The characteristic functioll of the 
set of T-free elenlents of E takes the form 
E, 
DT 
where O(T) = 4T) 
, k is the canonical additive character on E and,  defined in 
[Co], {k»  D  D} (where k(w) := k(w), w  E) is the set of all additive 
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characters of E of ordcr D (Dix ' - 1). Again, the integral notation represents a 
weighted sure: 
p.(D) 
DIg DIg (SD) 
Using these characteristic fimctions, we derive an expression for =(t, T): 
(2.1) ,(', )= 0<')(-)() iD   oe(5)(,C)  (,Oe(W)((D 
We shall now spccialise to thc case Whell ri = 3. Observe that, if p (i.e., if 
q = 3 k gor some k E N), theli 5[ = 1 and the FNT problem reduces to the 
(nonzcro) PNT problcln. If q  2 (mod 3), then I = x 2 + x + 1 is irreducible over 
; b)  llllll 3.50Ï [Co], $(lll, ]) > 0 i d only if =(. 1) > 0, and so the PFNT 
proBlcln rcduces to thc (nonzcro) PNT problcm iii this ce also. Hence onlv in 
the cc whcn q oe 1 (lllOd 3) lleed the fill PFNT problcm be considered. When 
q  1 (lnOd 3), [ = (x -- )(X -- 2) (whcrc    is such that ?a = 1,   1). 
With regard to thc multiplicative part of thc problem, we note that ail prime 
divisors of m must be congruent to 1 lllodlllO 6. r, SillCe ml(q  + q + 1), an odd 
lmmber, thcn m is odd. Frthcrnlore, suppose that for some prime I, lira. Then 
iq a - 1 but l  q- 1; hcnce ordq = 3. By Ferllat's Little Theorem. qt-  1 (mod l) 
since l  q. So 31/- 1. i.e., 1  l (nlOd 3). çhus all prinle divisors of m lie in the set 
Out strategy for proving the PFNT problem for n = 3 is to apply a sieving 
technique. We shall use the basic sieving inequality introduced in Theorem 3.1 
of [Co]. Let dira and ri x- 1. Then (di,L) (i = 1 .... ,r for r E N) will be 
called coplementary divisor pairs with common divisor pair (d0, f0) if the primes 
in lcm{d,...,d} are precisely those in d. the irredu«ibles in lcm{f, .... f} are 
precisely those in f. and for any distinct pair (i, j). the primes d irreducibles in 
gcd(di, di) alid gcd(f,fj) are precisely those iii d 0 and f0 respectively. Observe 
that the value of =(d, f) will depend only on which "atoms" (primes/irreducibles) 
are present in d and f, hOt on the power to which the atoms occur. 
Lemma 2.1 (Sieving inequality). For divisors d of m and f of x  - 1. let {(dl, 
.... (d, f)} be compleetary divisor pairs of(d, I) witfl common divisor (do, 
(2.2) (d, f)  ( (di, fi)) - (r- l)(do, fo).i= 
In the PNT ce, where there is no additive conlponent, the sieve will clearly 
take the following simpler form. For divirs d of m, let d .... , d be divisors of 
d (with common divisor d0) such that the primes in lcm{d ..... d} are precisely 
hose in d and, for any distinct pair (i,j), the primes in gcd(di, dj) are precisely 
those in d0. Then 
In the next section, we establish estimates for ( 1) ('m). 
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3. ESTIMATES FOR INTEGER FACTORS 

In this section we obtain new estimates for the number N(t, 1) of t-free elements 
of E with prescribed norm and trace, where t E N is a divisor of n. We improve 
upon the estimates of [Ci by applying some deep reslflts of Katz arising from the 
study of Soto-Andrade sums [Ka]. 

Lemma 3.1 ([Ka], Theorem 4). Suppose that n > 2. Then 

(3.1) N(1, 1) q(q- 1) 

_<nq , 

(3.2) Irr(1,1)-(q-l)l<n 1- q  

In particular, for n = 3, Lemma 3.1 has the form 
,Tr(1,1)-(q3--1),_<3(1--)q 
Note that this is an improvement, by a factor of approxinlately 3 , on the 
estinlate 
[zr(1, l)-q3,_< (1 (e+l))q 3q 
(e := gcd(3, q- 1)), obtainable from Corollary 2.2 of [Co] but useless as a lower 
bound. It is such increases in accuracy that allow us to solve the n = 3 case where 
the method of [Co] fails. 
Next, we estinaate N(t, 1) where tire, t > 1. 

Lemma 3.2 ([Ixa], Corollary of Theorem 3 bis). Let rl be a character of E of order 
d, where dira, d > 1. Suppose tf (q-1) is not trivial. Set 

Then 

I()=  

N(z)=b 
T(x)=a 

IAI()I _< nq  

This lemma is applicable when n = 3 to ail ?]d ¢ f* (dira, d > 1). For, consider 
some r/  /6. of order d, where dira and d > 1. Clearly r/q-1 cannot be trivial or 
have order 3, since (d, q - 1) = 1 and (d, 3) = 1. 

Corollary 3.3. Let tire, t > 1, and toit, to > 1. Then 

I o(t) r(to, 1) 
(3.3) rr(t, 1) O(to) 

( ) 
< nO(t)(W(t)- W(t0)) 1- q  

Pro@ By definition, 

N(t, 1)=O(t) 

 fa.trla(w) = O(t) falt M(rla)' 

N(w)=b 
T(w)=a 
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and so 

N(t, 1) 

By Lcnuna 3.2, 
N(t, 1) 
and hence 
(t, )- 
lu parti{'ulm for n = 3, 
Off) 
(3.4) -(t, 1} 

O(to) 

N(t0,1) = O(t) d/l 
O(t) N(to, 1)l < O(t)(W(t)- lV(to))nq  
O(t) - 

(«o, 1) < o(t)(u (t)- V(to)) 1- q  

<_ 30(t)(IV(t) - lI'(to)) ( 

4. TtlE (NONZERO) PNT PROBLEM 
Recall from Section 2 that, if q is a power of 3 or if q = 2 (mod 3), then the 
PFNT problem reduces to the (nonzero) PNT problenl ("nonzero" refers to the 
fact that the prescribed trace a is nonzero). Hence, to establish the result in these 
cases, it suffices to show that 7r(m, 1) > 0. 
Iii order to simplify uotation, ff'oto this point onwards we shall adopt the con- 
vention that ail UIlmarked summation signs have index i rumling fronl i = 
(where s is the nmnber of distinct primes dividing m), and that p[i] denotes the ith 
prime congruent to 1 modulo 6, i.e., the ith elelnent of the set {7, 13, 19, 31.37,...}. 
The following lemma provides a useflfl upper bound for 
Lemma 4.1. For any positive integer t, 
(4.1) W(t)  et Il/6, 
2  
where ct ---- (pl...p,,)/6, and Pi,... ,p are the distinct primes less than 64 that divide 
t. In particular, if pi  1 (nlod 6) for all i : 1,..., r, then ct ( 3.08. 
(The proof is obvious using multiplicativity.) 
Proposition 4.2. Suppose q is a prime power, q  1 (mod 3). Then (q, 3) /s a 
PNT paw for all q _ 622,346. In particular, (3 , 3) is a PNT pair for all k 
k>12. 

Proof. Apply the bounds of Lenlnla 3.1 and Corollary 3.3 directly, without sieving. 
Then 
rr(,n, 1)>_ O(m){(q3- 1)- 3 (l-)q- }-30(m)(IV(m)- 1)(1-)q, 
and so rr(m, 1) > 0 whenever 
( )1 
(4.2) q½ > 3W(m) 1 - +--. 
q2 
Using the approximation of Lermna 4.1 for B'(m), (q, 3) is a PNT pair whenever 
1 
(4.3) q > 3«.(q- 1) +- 
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where cm = 3.08. This inequality holds for intcgers q _> 622,346, and so esta.blishes 
the result. [] 

The following simplification applies in the case when 31q and m is prime. 
Lemma 4.3. Let q = 3 k, k E 1, so that m = q2 + q + 1. Suppose that m is prime. 
Then 
N(m, 1)=N(1,1), 
where N(t, 1) (tire) is the number of t-free elements of E with trace and norm equal 
to a and b respectively (a, b  F, a ¢ O, b primitive). 
Proof. Suppose a G E (i.e., trivially 1-free) with Tf(a) = a, N(a) = b, but  =/3 m. 
Then aq1 __ 1, i.e.,   GF(q). Hence, TrE/F(a ) = 3, which equals 0 since 
charF = 3, a contradiction since a # 0. [] 

Proposition 4.4. Suppose q is a prime power, q  1 (mod 3), and let m = 
pï ...p». Then (q, 3) is a PNT pair whenever 

(4.4) 

- }_(_),(, _') »0, 

and so certainly whenever 
(4.5) 
where 

( s-1 ) 1 
Cs:=3 2+ 1 s 1 +-, 
- ]= ,1 3  
where p[i] is the ith prime con9ruent fo 1 modulo 6. 

Proof. Apply the sieve with atonfic divisors. 
r(m, 1) > 0 whenever 
(,) -Æ- - 
By Lemma 3.1, (rn, 1) > 0 if 
(4.6) q >3 1- 1+ 

Using the results of Corollary 3.3, 

E(--1))  

(4.7) 
where 

Replacing the right-hand side of (4.6) by a larger quantity depending solely on s, 
we sec that the desired result certainly holds when 
1 
q > C 

( --1 ) 1 
+-. 
C:=3 2+l_]p_ 3 
Observe that, since C is a constant for fixed s and increases as s increases (for 
all s such that ]=lS [11 < 1), q½ > C1 for some si implies that q½ > C for all 
881.  
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Proposition 4.5. (i) Suppose q = 3  (k E N, k > 5 or k = 3). Then (q, 3) is a 
PFNT pair. 
(ii) Suppose q =_ 2(mod 3) and q < 622,346 but q  {5,8, 11,1ï, 23,29,32,47, 
53,107. 137, 149, 191}. Then (q, 3) is a PNTpair. 
Proof. (i) Lemlna 4.2 bas established the result for k > 12; se we need consider 
only k < 12. 
Let m = pï ...p8. We apply Proposition 4.4. For ail q = 3 a with k < 12 we 
bave s < 5. Sincc C52 < 577 < 36 , the result holds for q = 3 , k > 6. The result 
is established for k = 5 (s = 2), silwe 35 > 71 > C2 . When k =3,  (= 757) 
is prime; hence by Lemlna 4.3, m nmy be replaced by 1. Inequality (4.2) is then 
sa.tisfied, since vf > 2.8892. 
.. as. __ q2 
(ii) Forq> 2,1et m =pï .p , silwer < +q+l, s <8 forq<622,346 
(lnerely by size considerations). As in part (i), we apply Proposition 4.4. 
Since ilmquality (4.5) holds for ail relevant q > 1622, the result is established 
for prime powcrs q >_ 1637. For q < 1622, we find that s < 4; then the desired 
rcsult holds fol" q > 361, i.e., for all q >_ 367. Since the smallest q -- 2 (mod 3) with 
s -- 4 is q -- tq 9, use of inequalitv (4.5) with s 3 thon establishes the result for 
q > 204, i.e., q >_ 227. Howcver, evcn use of exact values fails for those q < 204 with 
s -- 3, namely (107, 137, 149, 191}. Similarly, (4.5) holds with s = 2 for q > 98, and 
thus establishes the rcsult for ail q _> 101 (apart frein the preceding exceptions). 
Use of exact values in inequality (4.6) yields the result for q -=- 83 (m -=- 19  367. 
 > 9.110 > 9.065 >right side of (4.6)). Values of q with s --- 2 for which exact 
values are insufficient are (11, 23.29, 32, 47, 53}. Finally, q½ > C for all q > 36. i.e., 
q >_ 41, which establishes ail remaining cases with the exception of (5.8, 17}. [] 

5. THE PFNT PROBLEM 
In this section, the full PFNT problem will be solved for the case when q = 
1 (mod 3). 
Denote by L a linear factor of M (= x 2 + x + 1); L lnay take the values x - 3' 
or x - 2, where  E F is such that 3 = 1, 3"  1. We begin by deriving estimates 
for the number N(1, L) of L-free elclnents of E with prescribed norm and trace. 
For economy of calculation, it is in fact desirablc to consider the difference between 
7r(1, L) and 0(L)Tr(1, 1) (in solne sense the "error tenn"). We will prove the following 
lemma. 
Lemma 5.1. Let q = 1 (mod 3). Then 
(5.1) 

( 3 
_<2q 1 q 

q) +2q2 (1- )  

First, we establish seine results about 5L. For a polynomial f(x), let f' denote 
the polynomial obtained frein f by replacing x i by x q' . 
Lemma 5.2. If Dix n/k - 1 (where kln ), then 5D is a roof of (x n/k - 1) , i.e., 
5D  GF(q/k) . 
Proof. Set R = qn/k. Se for the canonical chara.cter 
(w  E), where A(x) = e » Let \(w) = k(w) = A(TrR/r,(Sw)) and suppose 
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GF(R); so 5 n = 5. Then 

Hence \(w  - w) --- 1 for ail w E E. So for any Dix n/k - 1, i.e., D«lx  - x, 
k(D«(w)) : 1 Thus ( : (D for SOIlle Dix n/k - 1. Letting 6 vary in GF(R) 
accounts for all R characters of order dividing x'/k - 1. [] 

Lemma 5.3. Suppose q _= l(mod3), and let 7  GF(q) be such that 7 3 = 1. 
7#1. 
(i) Let D -= x - 7. Then (x - 72)a(SD) ---- 0, i.e., (»q : 725D . 
(ii) Let D = x - 2. Then (x - ")«(SD) ---- 0. i.e., (D q = "(D. 

Proof. (i) Suppose 5 q = ,25. Dcfine \(w) = Xl(SW) = A(Trqa/p(5W)), w e 
E = Fqa. Then 

(w' - 7) = (Trq/[Trq3/.(5(," -"/,))]) 
= A(Trq/v[Trq3/q("/Sqw q - 
= .X(Tr,,/,[ Tre/,,((,)" - ,,)1) 
= 1, 

since Tq3/q((Sw) q - 5w) E O. So the F-order of \" is x - . This accounts 
for all characters with F-order x - . 
(ii) Replace "7 by "7 2 in (i). [] 

We are now ready to prove Lenlma 5.1. Throughout this discussion, 
(where z] is a multiplicative character Oll Fq,) will denote a Gauss SUUl in 
We will use the notation J(oel,---, oea) (where a  F, oel .... , oek are umltiplicative 
characters of F, k  N) to denote the Jacobi sure 

 //1( C1 ) .. 
Cl +... +c-a 

For extra background nlaterial, the reader nlay consult text.s such as [LiNi]. 
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where  runs through ail (L) elements of A (i.e., : runs through ail dditive 
characters of E of order L). Separating the terre for which c = O, we have 

r(1, L) - O(L)r(1,1) 

(5.3) 

-- ql {,eP --" ï"(b) --" g'(w)k(SLw). (,L) 
vEF* c-F° (6L) 

For the first terre on the right side of (5.3), using the fact that L # 0, replace w 
by OE to obtain 

vF* () 

Since F* A D = AD, 

(') q- 1 -" -" ç(CSL)- - _, ç(SL) ç(c) , 
OL) cF" q 1 

and the inner sure equals 0 unless u* (:= 5IF) is trivial, when it equals q - 1. 
Notethat, for k E F, v*(k) = 5(k) = v(N(k)) = v(k3), i.e., v* = v 3. Sothe 
first terre of (5.3) can be simplified to 

For the second term on the right side of (5.3) (i.e., the part for which c -¢- 0), replace 
L by OSL, then w by «OÇ+I)' to get 

(5.4) 

'v () Ga()' /(SL+ 1)  X(ac)/(c). 
ver* (L) cF" 

Consider the inner sure 'ceF* X(ac)/(c) of (5.4); in the case when v a = ri, this 
reduces to a sum over additive characters of F, while for v 3 #- ri, a Gauss sum over 
F is obtained. Thus the second terre of (5.3) may be expanded as 

v ff* 
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HellC% 

rr(1, L) - O(L)rr(1, 1) = 

q 

(L) 

+  
vEF* 

5(JL + 1)) / 

(-) C,13(oe)-'[Oe(N((L ÷ 1))--(N((L))] 
(L) 

÷ l))dl (oe3)13(oe) 1 

since (73(5) = Gla(ee) by the Davenport-Hasse Theorenl (see [LiNi], Chapter 5). 
Consider the specific values that may be taken by L, namely L = x-3` and 
L = x - 3`2. Bv Lenlna 5.2, since these L are divisors of x 3 - 1, 5L q3 = 5L. Using 
Lenmla 5.2 and Lemma 5.3, we find that 5L 3 E 1Fq* but JL ¢ 1Fq*, and so JL 3 = c, 
where c is a non-cube in F. In fact, {5x-,} U {5x_,2} : {e E E: e 3(q-1) ---- 1, 
e (q-l)  1} ---- {cube roots of c, c a non-cube in F}, a set of cardinalit.y 2(q - 1). 
In the case when L = x - % using Lemnla 5.3 we get 

N((L) -= 5LSL qSL q2 = 5L(3`U(L )("SL ) = 5L 3 = C 

and 
N(1 ÷ aL) -- (1 ÷ 3' ÷ 3`u)(SL ÷ 5L 2) = (1 ÷ (L 3) = 1 ÷ c. 
The saine values are obtained when L = x - 3'2. 
Denote x-3' and x- 3 `2 by L1 and Lu respectively. Let, '3  ri* be an arbitrary 
character of degree 3. Then 
rr(1, L1) + rr(1, Lu) - 20(L)rr(1, 1) : _2 {$2 - S } 
q 

where 
() ( 1 ) 
(5.5) S 1 ":  g l(tZ3)Ç13(t z)  1 -- (tz3(C ) ÷ tz32(C)) #(1 
Ef* cE F* 

+c) 
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and 
(5.6) 

+«)-,«)). 

ISll <_ (q - 4)vfq(1 + 

cEF* cEF" 

cEF o 
+  .2(c)r,( +c-  .(c),(c)) 
cEF* 
1 
= (-1)-O-(Jl(U,)-(q-1)+Jl(U,)-O) 
1 J1 ( ug, 
= (q-2)-  

since voe = v 1 and v2oe : v. 
Thus [o I <_ ½(q - 2) + ½ v@ Hen«e, 

E 9(1+c)- E P(c) 

(5.7) }Sll_<2q{ (1-) Çl+qq). 
Now consider $2 (as given by (5.6)). For a given v with v3 = oel, oe -# oel, the 
inner sure a2 h the form 
(1 
cF* 
where v is an arbitrary character of order 3. Without loss of generality, we may 
oet oe := oe in our expression for 
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+ 2q 

The following is a sufficient condition for (q, 3) to be a PFNT pair. 

Lemma 5.4. Suppose q -- 1 (nlod 3). 
(5.9) 
(1, 1) (0(m)- ) 
> 30(m)(ll'(m)- l) (1- ) q + 2q (1 
Pro@ Apply the sieve in the following form: 

Then (q, 3) is a PFNT pair whenever 

7r(m,M) >_ r(m,1) + r(1,x-"ï) + 7r(1,.r-72) - 27r(1,1). 
Using the lower bomldS for 7r(m, 1) and the r(1, Li) (i = 1, 2) fronl inequalities 
(3.3) and Lemnla 5.1, we see that 7r(m, M) > 0 whenever (5.9) holds. [] 

Lemma 5.5. Let q = 1 (mod 3) be a prime power, and let m be the greatest divisor 
of q3 _ 1 co-prime to q - 1. Then 
1 
O(m) > --., 
q 
Proof. Observe first that, if l is a prime divisor of m, then l is congruent to 1 
11 
modulo 6 and hence l > 7. Since x-x - 1 > 0 holds for x > 7, it follows 
 > 1 where > 7isprime and k G N. Thus 
that O(p k) = O(p) -- p-1 > 1 _ r p _ 
P p12 (pk) 
1 Since 3m < q3--1 
by multiplicativity, O(m) > -----_. - q-1 < (q + 1) 2, it follows that 
-- 1 1 and so 0(m) > @ > 
q > 31/2m 1/2 1, and so q _> m for all q. Hence _1 < --_1, _ _ 
q -- Tt2 Ttl2 
-11 [] 
q 

Proposition 5.6. Let q = 1 (mod 3) be a prime power. Then (q, 3) is a PFNT 
pair for all q >_ 252,950. 

Pro@ By Lemma 5.4, r(m, M) > 0 if 

(5.10) 

-(1, 1) (0(m) - ) 
> 30(m)(V(m) - 1) (1 - ) 

q + 2q (1 q 3 
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Then by Lemnla 3.1, r(m, M) > 0 if 
(5.11) 
O(m) (q3 - 3IIr('m) (1- ) q - l) > 2q (1- 6 + -)q 

By Lemma 4.1, II'(m) _ < 1 c'q 1 , where cm < 3.08. 
1 
3W(m) _<  and so 3II'(m)(2)q < d(q- l)q 2 
«,n ' - " 
Lemma 5.5, we see that (m, BI) > 0 certainly if 

5 
Set d := 3c,; then 
Using this result and 

(5.12) --i-{q3-d(q-1)q2-1}>2q 1--+ +2q 2 2- , 
qg q q 

i.e., if 

(5.13) q>d(q 1)-+2q] (1 6 2 )  ( ) lq, 
- --+ +2q 2- +--. 
q 
Take c = 3.08 and .set d = 7.70 in inequality (5.13). Then inequality (5.13) holds 
for all q _> 252,950. 
Iu order to establish the result for smaller prime powers q. we will use the 
following suflïcient condition, which arises from the application of the sieve with 
atonlic divisors. 
Once again we shall adopt the convention that ail umnarked smnmation signs 
have index i running from i = 1 to s. 

Lemma 5.7. The following is a sufjïcient condition for (q, 3) to be a PFNT pair. 
When q _= 1 (nlod 3), 

(3s + 2) (3s+6) 4 3(1 - ] ± + 2 3 1 
(5.14) 
> 
1 2 q 
1 - - p, q 

(where m -= pÏ . . . p» ). 

Proof Let m = pî ...p», where pi ..... p are distinct primes and s G N (recall 
that the values of the ci will be irrelevaut here). Apply the sieve in the form 

(5.15) Tf(re, Iii) > 7r(pl,1)+...+Tr(p,l)+Tr(1,x-/)+Tr(1.x-7E2)-(s+l)r(1,1). 

Using the results of Lenlma 5.1 and Corollary 3.3, r(m, M) > 0 if 

(5.16) 
( l 
-(1, 1) 1- E p i 
- 2q (1 q 3 

>0, 

i.e., if 

(5.17) r(1,1) > 

q  ((3s + 2) (3s+6) 4 
q q2 

3(1-)E±)p, + 2q2(1_ )3 

1 2 
1 --Ep, q 



PRIMITIVE FREE CUBICS WITH SPECIFIED NORM AND TRACE 3113 

and so, using Lemnm 3.1, certainly if 
(5.1S) 
v((3s + 2) (38+6) 4 ' ± + 2(1 - ) + 3/(1 - }) + 1. 
q> q 
1 2 
1 -Ep, 
Observe that the inequalities of Lemma 5.7 are of use only when the denominator 
 < 1. However, since 
 a > 0; in particular, it is necessary to have 
1 -p q 
all prime powers q that are congruent to 1 modulo 3 and less than 252,950 have 
s  7, and all prime divisors of m are congruent to 1 modulo 6, the denominator is 
always poeitive in this case. 
Proposition 5.8. Suppose q  1 (mod 3) and q  252,950, but q  {7, 13, 16, 19, 
Then (q, 3) is a PFNT pair. 

25, 31,37, 43, 49, 61,64, 67, 79,109,121,163,211,256}. 
Proof. For q > 4, observe that 
3 
1> q2 
±> 3 3(1_1 1 
since  p, _ q2+q+l -- - q -- q(q+q+l))" 
5.7, the desired result holds if 

3 
Using this lower bomld in Lenlnm 

(5.19) / > q 
An upper bound is required for 
Pi  Pt -- 
genernl, to simplffy calculntons, fle crude estmnte 
s s 
(5.20, 
=Pi .=P 

hl 

will be used, where p[i] is the ith prime congruent to 1 modulo 6, as in Section 4. 
(More precise values may be taken in specific cases.) 
Observe that the desired result certainly holds when 
2 18 
(3s + 2) +  + oe 1 
1 2 + 3 +--, 
(5.21) v/ > 1-  p q q2 
and, for fixed s, the function of q on the right side of (5.21) clearly decreases as 
q increases. Hence to prove for a given s that the result is true for q > q0 (some 
q0 E N), it is sufficient to show that inequality (5.21) holds for q = q0. 
For q < 252,950, we bave s < 7. Using the basic estimate 
1<1 1 1 1 1 1 1 
(5.22) E pï -  + - + Ï +  +  + - +  < 0.3714, 
inequality (5.21) holds with s = 7 for relevant q > 1580, hence for ail q > 1597. 
Now, for prime powers q = 1 (mod 3) less than 1580, it happens that s _< 4; in fact, 
except for the two values q = 919 and q = 1369, s _< 3. Using the estimate 
1 <1 1 1 1 
(5.23) E p-7 -  + - + - +  < 0.3047, 
inequality (5.21) holds with s = 4 for q > 546 and hence for all q _> 547. For 
s = 3, use of inequality (5.20) in (5.21) establishes the result for q > 339, i.e., 
q _> 343. For q = 277 (m = 7- 19- 193) and q = 289 (m = 7- 13. 307), use of exact 
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values in Lemum 5.7 establishes the result However, this approach is insufficient 
for {121,163,211,256}. In the s = 2 case, inequality (5.21) establishes the result 
for q > 185, i.e., q > 193, when applied with the a.piroxinmtiol of (5.20), aud for 
q = 169 (m = 61 -157) and q = 181 (m = 79- 139) when exact values are used in 
(5.21) (respectively, 181 > 153.49 and 169 > 148.80). Use of Lemma 5.7 suffices 
for q = 139 (.m = 13-499), since 139 > 137.14. Outstanding exceptions in the 
s = 9, case are { 16, 25, 37, 49, 61,64, 67, 79. 109}. When s = 1, replacing pi by 7 in 
inequality (5.21) establishes the result for q > 86, i.e., q > 97; use of exact Pi (= m) 
deals with the case q = 73 (m = 1801). The relnaining exceptions with s = 1 are 
{7,13,19,31,43}. [] 

6. COIVlPUTATIONAL STRATEGY FOR REMAINING CASES 

To deal with the 34 cases relnailfing a.fler Propositions 4.5 and 5.8, we use the 
computer package hIAPLE (version 6) to search the field E for m-free elements with 
norms and traces equal to the required values. (For reference, the set of exceptional 
q is as follows: {3,5, 7,8,9,11,13,16,17,19,23,25,29,31,32,3ï, 43,4ï, 49,53,61, 
64, 67, 79, 81,107, 109, 121. 137,149, 163, 191,211,256}.) 
The following lemlna allows us to silnplify our COlnputatiolml strategy in some 
cases for which the PFNT problem reduces to the PNT. 

Lemma 6.1. Let q be a prime power, q  1 (mod 3). Denote by Zc,,(m) the 
number of eleme,ts w G E that are m-free and bave TrE/F(W) = Ct. NE/F(W) =  
(,  e F). Suppose 

Z,b(m) >0 Vb  F*. 

Then (q, 3) is a PNT pair. 

Pro@ To prove tha.t (q, 3) is a PNT pair, we must show that N(m, 1) > 0, i.e., 
that Za,b(m) > 0 for ail a, b  F, a 7 O, b prinfitive. Vre prove the (stronger) result 

Z«,b(m) > 0 Va, b  F*. 

If a = 1, there is nothing to prove. Otherwise, set b* --'- ab G F*. Since Z,b* (m) > 
0, there exists au element ç  E such that ç is m-ri'ce, TrE/,(ç) = 1, and NE/c(ç) = 
b*. Then c := aç is also r-free, and has TrE/é'(c) = a and N:/:(o) = b. 
Use of Lemlna 6.1 reduces the number of necessary tests froln (q - 1)¢(q - 1) 
(testing each pair (a, b), b prinfitive) to q - 1 (testing each pair (1. b), b nonzero). 
Since the condition involved is stronger than the PNT condition, this silnplification 
is only of practical use in those cases when q - 1 is prime, or ¢(q - 1) is not too 
much snaller than q - 1. However, it is successful in dealing with ail q  1 (mod 3) 
up to q = 32. For larger values of q, we nmst search E explicitly. 
In the PNT case, the desired result holds without exception for ail q  1 (mod 3) 
remaining from the previous sections. 
As an illustration, we display the relevant cubic polynomials for the case when 
q = 5. The following table lists eight cubic polynomials over F = GF(5) whose 
roots c G E = GF(5 a) are primitive and free with norm and trace equal to b and a 
respectively. 
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(a, b) Relevant PFNT cubic 
(1,2) x 3 + 4x 2 + 3 
(1,3) x 3 + 4x 2 + x + 2 
(2,2) x 3 + 3x 2 + 2x + 3 
(2,3) x 3 + 3.r 2 + 2 
(3,2) x 3 + 2x 2 + 3 
(3,3) x 3 + 2.r 2 + 2x + 2 
(4,2) x 3 + x 2 + x + 3 
(4,3) x 3 + x 2 + 2 

The cubic polynomials given in the table for (a, b) = (1.2) and (4.3) are in fact 
unique. Thus, when q = 5 and n = 3, we observe that in some sense the PFNT 
property "only just" holds. 
In the case when q = 1 (mod 3), we search through E explicitly for elements 
with the required properties. The following lemma lets us reduce the nunlt)er of 
1 
pairs (a,b) that must be tested from (q- 1)¢(q - 1) to g(q - l)¢(q - 1). [] 

Lemnla 6.2. Let q = 1 (mo(l 3), and set k 
ist ffee, primitive c E E such that 'E/(c) 
pairs (a,b) where b is a primitive element of 
/3 a fixed primitive element of F}. Then there 
that TrE/,(c) = a and NE/(c) = b. for all 
element of F and b is a primitive element of F. 

 -- q- Suppose that there ex- 
= a and NE/oe(c) = b. for ail 
exist free, primitive c  E such 
pairs (a. b) where  is a nonzero 

Pro@ Fix a primitive element fl of F. Observe that F* nmy be partitioned 
nto k cosets of the subgroup H := {1,flk, 2} of cube roots of unity, namely 
H, flH,...,2H,...,fie-tH. The result follows since TrE/F(h3,) = b TrE/F(3,), 
N(hT) = b3NE/F(3 ,) for all 3'  E, h  F. 
Without exception, for ail q -= 1 (mod 3) remaining from the previous section. 
(q, 3) is found to be a PFNT pair. 
In closing we remark that, for each of the larger values of q amongst the set of 
exceptions, the computations to check all the possibilities took several hours to run. 
(In the case of q = 256, the original computation time of several weeks was reduced 
to a few hours by reprogramming.) This vindicates the efforts we have ruade to 
solve the problem theoreticallv in as many cases as possible. [] 

[Cal] 
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[CoHu2] 
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D-LOG AND FORMAL FLOW 
FOR ANALYTIC ISOMORPHISMS OF N-SPACE 

DAVID WRIGHT AND WENHUA ZHAO 

ABSTRACT. Given a forrnal map F = (F1 .... , Fn) of the forrn z+higher-order 
terrns, we give tree expansion forrnulas and associated algorithrns for the D-Log 
of F and the forrnal flow Ff. The coeflïcients that appear in these forrnulas can 
be viewed as certain generalizations of the Bernoulli nurnbers and the Bernoulli 
polynomials. Moreover, the coefficient polynomials in the forrnal flow formula 
coincide with the strict order polynornials in cornbinatorics for the partially 
ordered sets induced by trees. Applications of these forrnulas to the Jacobian 
Conjecture are discussed. 

1. INTRODUCTION 

This work began as an effort to link and extend the results of [W2] and [Z], 
placing them in a common framework. Both of these papers deal with the formal 
inverse F- 1 of a system of power series F = (Ff ..... F,); bot h give formulas for F- 1 
in terres of F, the former being a tIee formula, the latter an exponential forImlla. 
This quest bas led to a host of interesting COlmections, algorithms, fornmlas, and 
rela.tioIships with combiIatorics, Bernoulli nuInbers, a.nd Bernoulli polynomials. 
The forIner paper dcals with tree fornmlas as they apply to formal inverse, 
a thread which is also the main thrust of [BÇW], [,V1], [W2], and [ChIT\VW]. 
It has combiuatoric colmections with genera.tiug functious and enuIneration tech- 
niques for trees. The general goal of power series inversion (soInetimes called 
"reversion", perhaps to distinguish flulctional inverse from multiplicative inverse) 
is as follows. Let F = (F1,...,Fn) with Fi E C[[Zl .... ,z,]] for each i and 
Fi = Zl + terIns of degree > 2. One seeks forInulas for the unique G1 ..... G, E 
C[[Zl ..... z,]] for which G(F) = zi, for i = 1,...,n. Perhaps the first of these 
was the Lagrange Inversion ForImlla (see [St2], Chapter 5), which dealt with the 
case n --- 1, and which was generalized (under a certain restrictive hypothesis) to 
all n by I. J. Good [Go] in 1960. Good then uses his fornmla for problems of enu- 
merating certain trees. In fact, Good's formula had been discovered and published 
by Jacobi in 1830 [Ja]. Another paper which appeared in 1960 was that of G. N. 
Raney [RI, who also related forInal inverse to trees. Ra.ney's work was generalized 
in [ChITWW], which also utilized the work of Jacobi. A general inversion formula 
was given by Abhyaukar and Gurjar in 1974 lAI, and this is the source from which 
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the tree fornmla of [BCW] was derived, with the hope of applying it to the Jacobian 
Çonjecture. Other treatments of the subject of inversion are [HS], [Ge], and [Jo]. 
The tree formula of [BCW] expresses the formal inverse F -1 as an infinite - 
linear combination of certain power series JT ( C[[z]], which are constructed using 
finite rooted trees T. This coustruct.ion will be reviewed in §2 and a new (and 
quick) proof of the inversion fornmla, using the tools developed in this paper, will 
be present.ed in §5 (Theorem 5.1). 
Amongst the results of the latter paper is the realization of F by an expression 
F = exp(A) .z, where A = A(z), called the D-Log of F, is a differential operator 
uniquely determined by F a.nd yielding the fOrlnal inverse as F -1 = exp(-A)  z. 
Furthernlore, the formal flow Ff = exp(tA)  z encodes ail powers F [n] with   7/, 
of the formal map F. The D-Log and thc formal flow will be reviewed in §3. 
A primary goal was to show that the D-Log A can also be expressed as a Q-linear 
combination of the power series [PT. This goal was attained, yielding a tree formula 
for the D-Log. IXloreover, we discovered that the rational coeflïcients bT of this 
expression can be generated by an elegant recm'rence relationship and possess some 
intriguing combinatorial properties. I%r example, the Bernoulli numbers appear 
amongst these cocfficients. 
This situation is placed in a larger context which iucorporates formal inverse 
by considering the formal flow Ff = exp(tA)  z, where t is an indeterminate. For 
ri G iN, sett.ing t =-  gives the rt-fold composition F o ... o F, and setting t = -n 
gives the ,-fold composition F -1 o ... o F -. The system Ft can be written as a 
Q[t]-linear combination of the power series [PT, producing for each root.ed tree T a 
polynomial bT(t) having )T as the coefficient, of t. Among these polynomials are 
the binomial polynonfials (tre), for ail positive integers m. V'e give an algorithm 
for calculating bT(t) using the difference operator A. This formula is used to 
establish the relationship of certain bT(t)'s with the Bernoulli polynomials Bru(t) 
via an integration formula. It is shown that bT(t) also provides an interesting 
combinatorial connection: It coincides with the strict, order polynomial t(P, t) (see 
[Stl], Çhapter 4) for P = T, which, for t = m G 7/, +, counts the nulnber of strict. 
order-preserving maps from an 5' partially ordered set P to the totally ordered set 
with m elements. 

Acknowledgments. We would like to thank Professor John Shareshian for inform- 
ing us of Theoreln 4.5, and also Professor Steve Krantz for a helpflfl conversation 
on flow of analytic maps. 

2. TREE OPERATIONS 

2.1. Notation. By a rooted tree we mean a finite 1-connected graph with one 
vertex designated as its foot. The 1-connectivity provides the notion of distance 
between two vertices, which is defined as the number of edges in the unique geodesic 
connecting the two. The height of a tree is dcfined to be the maximum distance 
of an.v vertex from the root. In a rooted tree there are natural ancestral relations 
between vertices. Vv'e say a vertex w is a child of vertex v if the two are connected 
bv an edge and w lies flrther from the root t.han v. Iii t.he saine situation, we say 
v is the parent of w. Note that a vertex may have several children, but only one 
parent.. The foot is the only vertex with no parent. A vertex is called a leaf if 
it has no children. When we speak of isomorphisms bet.ween rooted trees, we will 
always mean root-preserving isomorphisms. 
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With these notions in mind, we establish the following notation. 
(1) We let "iI' be the set of isomorphism classes of all rooted trees and, for ru _> 1 
an integer, we let "iI', be the set of isomorphism classes of all rooted trees 
with rrt vertices. The latter is a finite set. 
(2) For any rooted tree T, we set the following notation: 
 rtT denotes the root vert.ex of T. 
 E(T) denotes the set of edges of T. 
 V(T) denotes the set of vertices of T. 
L(T) denotes the set of leaves of T. 
v(T) (resp. l(T)) denotes the mmiber of the elements of lr(T) (resp. 
L(T)). 
h(T) dcnotes the height of T. 
 CT denotes the numbcr of the elements of the autolnorphism group 
Aut(T). 
 For v E l'(T) we denote by aT,v the size of the stabilizer of v in 
Aut(T). Similarly, for e E E(T), we write CET,, for the size of the 
stabilizer of e iii Aut(T). 
 the two (distinct) vertices that 
For e  E(T) we dcnote by ve alld Ye 
connected by e, v«ith ve being the one closest te the root. 
 v  I'(T) we denote by v + the set of vertices that are children of 

are 
For 
For 

 v  V(T) we define the height of v te be the number of edges in 
the (unique) geodesic COlllleCtillg v te rtT. The heigbt of T is dcfined 
te be the inaxilllunl of the heights of its vertices. 
 For Vl,...,v  V(T), we write T\{vl ..... v} for the graph obtained 
by deleting each of these vertices and ail edges adjacent te these ver- 
tices. 
(3) A rooted subtree of a rooted tree T is defined as a commcted subgraph of T 
containing rtT, with rtT, = rtT. In this case we write T  <_ T If T  - T, we 
write T  < T. If T' < T, we write T\T' for the graph obtained by deleting 
all vertices of T' and all edges adjacent te its vertices. 
(4) For any k _> 1, we denote by Ck the rooted tree of height k - 1 having k 
vertices, and by Sk the rooted tree of height 1 having k leaves. We also set 
S0 = o, the rooted tree with one vert.ex. We refer te the trees C as chains 
and the S as shrubs. 
2.2. Power Series Given by a Rooted Tree. Let CI[z1 .... , z,]] = C[[z]] denote 
the ring of formal power series in n variables zl .... , z, over the complex munbers 1 
C. For i = 1 , n we will write D for the differential operator ô The operators 
D1,..., D. are cornmuting derivations acting on the ring C[[z]]. 
Given a vector of power series F = (F1,...,F) E C[[z]] ', we write Fi = zi + H 
for i = 1,...,n, or just F -- z + H. 2 In most applications the power series H = 
(H1,..., H,) will involve only monomials of total degree 2 and higher, and we will 
often take H te be hornogeneous of degree d > 2. However, these assumptions are 

lin this paper C can always be replaced by any -algebrm 
2We should here acknowledge that in almost every other treatment of this subject the system 
F is written as z - H, which yields nicer looking formulas for the formal inverse of F. The reason 
for our choice is that the formulas involving the D-Log and formal flow, which will be developed 
in §3, corne out better when we write F = z + H. 
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hot necessary for what follows here. We will associate to each rooted tree a power 
series in n variables based on F (equivalently, on H). 
For T  "lI', a labeling of T in the set {1,...,n} is a fllnction f " V(T) 
{ 1 ..... n}. A rooted tree T with a labeling f is called a labeled rooted tree, denoted 
(T, f). Given such, and given F = z+H as above, we make the following definitions, 
for v  V(T): 
(1) H,, = Hl(v); 
(2) 
(3) 
(4) Pr, y = H,,eV(T) D,,+ H,,. 
Finally, we define systems of power series PT = (PT, ..... PT,) and TT = (TT,,..., 
TT.) by summing over ail labelings of T lmving a fixed label for the foot 
PT, = 
I(rtr)=i 
1 
T.i = -- PT,i 
T 
for/= 1, .... 
One notes that the systems of power series PT and TT are dependent on the 
integer 
objects which detennine flmctions C[[q .... , z,]]"  C[[zl .... , z]]  for all n E 1. 
We will write Pç(H) and TT(H} when we need to emphize this dependence, or 
when we are dealing with more than one system H. 
2.3. Stable Linear Independence. We begin by establishing an important in- 
depcndence property of the objects {PT ]T  T}. 
Definition 2.1. We sw that rooted trees T ..... Tk are stably linearly dependent 
k 
if there exist c,..., ck  C such that ,= ciPç, = 0 for any integer n  1 and 
any homogeneous polynomial system H = (H,..., H,,) in n variables. Otherwise, 
we say that Ti are stably linearly independent. 
Remark 2.2. If H is homogeneom of degree d and if T  T, then Pç(H) is 
homogeneous of degree (d - 1}m + 1. Thus if we ptition {T,..., T} according 
to the number of vertices in a tree, then T,..., Tk are stably linearly independent 
if and only if each partition is a stably linearly independent set of trees. 
Lemma 2.3. Suppose that i= CiPT(H) = 0 lot any integer n  1 and any 
k 
homogeneous polynomial system H in n vamables. Then i= c{Pr,(H) = 0 for 
any system of power series H = (H,..., H) in n variables. 
Proo]. We first prove it for any polynomial H (hOt necessarily homogeneous) in n 
variables by introducing a new variable z,+ and homogenizing H using Z,+l. Call 
the resulting homogeneous system . Setting  = (. Hn+l = 0), we have 

k k 
z=l 

=0, 

which proves the lemma for H a polynomial system. For an arbitrary system of 
power series H we note that if T is a tree with r edges, the homogeneous summands 
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of degree < d in PT(H) depend only Oll the hoinogeneous sunmmnds of H having 
degree _< d + r. Taking r to be the lllaxillltllll of the nulnbers of edges iii T1 .... , Tk, 
then ail terres of degree <_ d in -=1 ci.PT, (H) depend only the hoinogeneous sum- 
mands of H having degree <_ d+r. Taking  to be the polynomial truncation of H 
of degree d ÷ T, we see t.hat Ei=l ciJDT (H) aild Ei=l ciJDT () coincide up through 
degree d. Since the latter is zero ( being a polynomial systeln) and d is arbitrary, 
we must have =1 cPT, (H) = O. [] 
Theorem 2.4. Any rooted trees T (i = 1,2 ..... k) with T.  Tj for any i # j are 
stably linearly independent. 
Before giving the proof we will define a polynoinial system depending on a rooted 
tree. Given a rooted tree T with fa vertices, we create variables Zl .... , zm. Label 
the edges e2,...,era and assign each variable z with 2 <_ i <_ m to the edge 
Label the vertices as follows: vl -- rtT, and for i ---- 2 .... , m let v be the vertex of 
es that is farthest from the root For each vertex vi  V(T), we define H to be the 
product of all the variables assigncd to the edges comlecting vi with its children. 
(Thus if ri is a leaf, we have H -= 1.) Set HT -= (H1,..., Itm). ¥e bave 
Lemma 2.5. Let T and T  be two rooted trees witl tle saine number of vertgces. 
Then 
(0 O) if TT', 
PT,(HT)= ((CT,O,...,O) ..... if T--T '. 
Proof. The following facts are hot diflïcult to verify and provide a skçtch of the 
proof: Each coordinate HTd of HT is a monomial that is linear or constant with 
respect to each variable ci. Each coordinate is constant with respect to . Each 
variable zi with i > 2 appears in precisely one coordinate HTj , and  # . P, (HT) 
is a homogeneous system of degree zero, and lnuSt be equal to either 0 or l. If a 
labeling f  V(T') --, {1,..., fa} is hot bijective, then PT'J = 0 since it would entail 
differentiating two different coordilmtes HT.  and HTj wit.h respect, to the saine 
variable, or differentiating some HT,i twice by the saine variable, or differentiating 
some HT. by zi, all of which give zero. Moreover, if f(rtT, ) # 1, then PT'd = O, 
since it would entail differentiation by Zl, and therefore PT,(HT) is zero except 
possibly in the first coordinate. 
With this it is not hard to show that, if f  V(T') --, {1,... ,m} is a labeling for 
which PT'j  O, then the function V(T') --, V(T) defined by w  vj() gives an 
isomorphism of ç  T  - T. Finally, the group Aut T acts freely and transitively on 
the set of labelings f  V(T) --, {1 ..... rn} for which PTj  O. The lemma follows 
easily from these statements. [] 
k 
Proof of Theorem 2.. Suppose that }-i=l ciPT, (z) = 0 with Cl 
HTx. Then there must exist j  1 such that PT (HT)  0. By the lemma above, 
we have T1  Tj. [] 
If H = (H1,. ., H) is a system of power series such that each Hi has only terres 
of degree d and higher, the power series TT has only terres of degree (d - 1)v(T) + 1 
and higher. Hence if d _> 2, a sure of the form }-TeV CTTT nmkes sense, since only 
finitely many terres contribute to any specified honmgeneous sunlmand. With this 
observation, we state the following consequence of stable linear independence. 
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Corollary 2.6. Suppose we bave a collection {CT} C C indexed by the rooted trees 
T E OE such that ETEç CTPT : 0 for any integer n >_ 1 and any system of power 
series H = (H ..... H,) with H having only terres of degree  2. Then CT = 0 for 
all T  OE. 
Proof. We consider systems H that are homogeneous polynomial systems of degree 
d  2. In this ce TT is homogeneous of degree (d-1)v(T)+ 1. So the homogeneous 
summands of Teç CTTT are the finite sums eu CTTT for N G N; so theoe must 
be zero. By Theoem 2.4 applied to the finite set of trees OEN, we must bave CT = 0 
for ail T  OEN- 
eca.ll that we arc writing D for the operator .  will denote by D the 
cohllllll vector (D, ..... Dn) t. b now define a differential operator on CI[z]] for 
each T G OE. 
Definition 2.7. For T G OE, we denote bv Dr the differential operator PrD = 
n 1 
i=1 PT,iDi.  will write r for the operator TD = 
2.4. ee Surgery.  will now discuss some "surgical'" procedures on trees. 
Given T ff T and e G E(T), the removal of the edge e from T gives a discon- 
nected graph with two COlllleCted COlllpOllellts which m'e trees. Xb denote bv Te 
the COlllpOll(,llt, colltaining rtT, and by T the other COlllpollçllt. We give T« and T 
the structure of rooted trees by sctting l'tT = rtT and rtT = v e. 
Given rooted trees T and T t a.nd v  I'(T), we denote by 
T   T 
the tree obtained by conlmcting riT, and v by a newly created edge, and setting 
rt(T,,.T) = rtT.  will refer to the newly created edge  the connection edge 
of T   T. Note that for any tree T and edge e  E(T) we have an obvious 
isolnorphism T  (7 , T¢) which is the identity on T« and . 
Given e, f  E(T), we say "f lies below e", and write e » f, if f e E(T«). This 
merely says that f remains when we "strip awoEv" e md T. One can eily see that 
this relation is hot transitive. However, if we write 
1  "'"  er, 
for e .... ,  E(T), we will mean by this that e  ej if i < j. 
A sequence ê = (ei ..... e)  E(T)  with e  --.  e dctermines a sequence 
of subtrees T¢.,. Tê,+ as follows: Set Tê3 = T  and let $2 = Te noting that 
e2,..., e  E(S2). For i = 1,..., r, assume that ¢,, .... Tê,_, Si are defined with 
ei,..., e  E(Si). Set Tê, i = (Si)C and Si+ = (Si)e. Finally, set Tê.+ = Sr+. 
For any integer r  ] and T  OE, create an indeterminate ). Denote this set 
of variables (for all T and r) by ]'. Extend the action of the operators DT and 
to C[[z]][Y] by making each il,deterlninate of t" a constant. 
Lemnm 2.8. Let r, m  1 be inte9ers and S  OE. Then 

( T ..... Tr )E'r r 
v(T1)+..-+v(Tr)+v( S)=m 
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Proof. Note that both sums are finite. 
H E C[[z]] '. 
We first consider the case r = 1. For T' E "F we have 

Hence 

So the expression makes sense for anv 

DT'PS = Z P(T'--%S). 

:EX Z 

-(1) D 
} T' I (T'--oS)  

TT, T'T vV(S) 
( T'--%.S)T 
For a fixed T G . we wish to count the occurrences of PT in the last expression. 
Toward this end, for T   , let 
IT.T,,S = {o e I'(5')  (T' v S)  T}, 
JT, T',S = {ê  E(T)/Aut(T)ITé  T',T  S (for anv e in ê)}. 

We will define a fmlction  : IT, T,,S "-- JT.T',S aS follows: Given v  IT,T,,S , 
choose an isomorphism ç  (T' v S)  T. and let e be the image under ç of 
the commotion edge in T' v S. Letting ê be the cls of e in E(T)/Aut (T), we 
clearly have ê  JT, T',S. To sec that ê is independent of the choice of ç, suppose 
: (T' v S)  T sends the connection edge to f 
hence f = ê in E(T)/Aut (T). Therefore, we bave a well-defined flmction , which 
is obviously surjective. 
We claire that for v  Ir.T,,S the orbit of v under Aut (S) is precisely the fiber 
ofv under . It is clear that ifw  v bv the action ofAut(S), then (T' 
S)  (T' v S)  T, with the first isomorphisn taking one connection edge 
to the other, which shows that w  IT, T,,S. Choosing appropriate isomorphisms 
(T'  S)  (T' v S)  T, we see that the image e of the connection edge 
of T' v S under ç is also the image of the connection edge of T 
hence (w) = (v). Moreover, ifw E IT, T,,S is element for which (w) = (v), 
then we have isomorphisms 
(T'S)T(T'v S) 
such that the same e E E(T) is the image of both connection edges. (This can be 
oehieved after modi-ing by an autonorphism of T.) It follows that y-ç : (T' v 
S)  (T'  S) carries one connection edge to the other; so it restricts to an 
automorphism of S sending v to w. Hence w  v. Therefore, the above smn tan 
be written  
v(1)p 
T¢., T¢ V¢IT.T.S 
T@m T' @ 
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TTm êEE(T)/Aut (T) 
Te_S 
where STe (ve) is the orbit size of ve under the action of Aut Te, for some (any) 
e E E(T) representing ê. The number of edges representing ê is OT/aT, e. Hence 
the }nner sure can be altered to run over all e  E(T) at the cost of dividing by 
T/T,e, yielding 
T 
T6T e6E(T) 
An automorphism of T fixing e  E(T) restricts to an automorphism of T é and an 
autonmrphism of T« fixing ve. Çonversely, given the latter pair we get a unique 
a.utomorphisn of T preserving e. [t follows that aT, e = TT,v«. AI we have 
Incorporating these facts aud putting together the above 

ST«,v« = CiTe/CTe 
equalities, we get 

E i:(1 ) 1 -(1) 
T'ET TET, eEE(T) 
,,( T' ) ÷ v( S )=ra Te--S 
Dividing the equation bv as and substituting ]) for 1) for each R  T vields 
v(T')+v(S)=m T«S 
whi«h is precisely the sertion of the lenmm for r = 1. 
For r k 2 we apply induction  follows: 

Z 
(T1 ..... 
(T)+...+(T)+(S)=m 

: E 
T T 

Z 
(T ..... T)ET - 
v(T)+...+v(T,-)+v(S)=rn-v(T,) 

-C 2) .(') 

Applying induction and a substitution of variables Yt (i+1) for }(i) to the inner sure, 

this equals 

E x:(1)O' 
T 2"T 

= E 
T1 
v(T, )+v( R)=ra 

=Z 

E 
TIT 
v(T)+v(R)=m 
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Now we apply the case r -- 1 to the bracketed expression to obtain 

T6T, 

T6T, 

ê=(e ..... er_)E({) r- 
1 ''" "»*r-- 1 
ê=(e ..... e)eE{T)  
T«,r+I--S 

which colnpletes the proof. 

Suppose the system of power series H = (H1,..., H,) has the property that each 
Hi involves only UlOnOnlials of degree >_ 2 in zl,..., zn. Then one easily verifies 
that for T  "1I', TT involves only mOlmlnials of degree _> v(T) + 1. It follows that 
for a monomial /lI in z of degree m, ©T  /lI involves only monomials of degree 
>_ m + v(T). Therefore, infinite sums such as Te" TT and Te" ©T make sense 
in this situation. The following two corollaries of Lemnla 2.8 are based on this 
observation. The equations in both corollaries take place in the ring C[Y][[z]], 
where Y represents the infinite set of variables {) [ T  "1I', i  Z+}. 

Corollary 2.9. Suppose that the system of power series H involves only monomials 
of degree >_ 2. Let r >_ 1 be an integer and S  "IF. Then 

(Tf ..... T)e'f  

Pro@ We simply sum (2.1) over ail m _> 1, noting the convergence of the sums by 
the observations above. [] 

Corollary 2.10. Suppose the system of power series H involves only monommls 
of degree >_ 2. Let k >_ 2 be an integer. Then 
E [v(1),-r, [v(- ),-,-, () 
, ,1--. ][< 
[ITk_ 1 Tk-I 
( Tt ,...,T ) eç  
T ê=(e,...,ek_l)E(T) - 
v{T)2 el---ek-i 

v ('+1) setting 
Pro@ We apply Corollary 2.9, multiplying both sides of (2.2) by "s , 
k = r + 1, summing over ail S  "1i". Note that the singleton tree contributes 0 in 
(2.2) for any r >_ 1, and thus the qualifier v(T) _> 2 in (2.3). [] 
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3. D-LoG AND FOHMAL FLOW 

We will henceforth be restricting our attention to systems of power series F = 
(F1,..., F?2) E C[z]] n of the form Fi = z + H, with H illvolvillg only lllOllOlllials of 
degree 2 and higher, for i = 1 .... ,7. \Ve refer to this condition by saying "F is of 
the forln identity plus higher." Such a system determines a C-algebra automorphism 
of C[[Z]], lla.lllely t|le a.lltOlllorphisill that Sellds z i to F i for i = 1,...,n. 

3.1. The D-Log. The following proposition appears as Proposition 2.1 in [Z]. 
Proposition 3.1. For any F = (Fx, F2,..., F,) E CI[z]] ' of the form identity plus 
higher, there exists a unique system of power series 
involving oTdy monopdals of degree 2 and htgher such that. lettmg .4 = aD = 
-i= c.iD, we bave 
exp(A)- z = F 

(3.1) 
where 

 A k 
and z = (Zl .... , Zn). 
The reader will eily verih" that the infilfite SUln exp(A)  Q lnakes sense for any 
Q G Ç[[z]]  due to the fact that, for any integer d  0. only finitely many terres 
A  
k " Q contribute to the degree d homogeneous sumnmnd. This is due to the fact 
that a involvcs onlv terres of degree 2 and higher. 
Remark 3.2. It is well known that the exponential of a derivation on any Q-algebra. 
when it lnakes sense, is an autolnorphisln of that algebra. Any subring lying in 
the kernel of the derivation will be fixed bv this automorphism. It follows from 
this fact, the comment above, and Proposition 3.1 that exp(A) is the Ç-algebra 
aUtOlnorphism of Ç[[z]] that sends zi to Fi, or i = 1 ..... 
Definition 3.3.  call the unique system of power series 
obtained above the Diffential Log or D-Log of the formal system F. 
3.2. Coefficients eT of the D-Log. 
Theorem 3.4. There exists a unique set of rat.tonal numbers {eT} indexed by the 
set of rooted trees T  T, such that 
(3.2) a =  
T6ç 
These numbers satisfy, and are uniquely deteined by, the following properties: 
¢ = 1 when v(T) = 1 (i.e., T = , the singleton tree), 
"() 1 
(3.3) ¢ = -  ï  ¢, ¢, . . . ¢«, 
k=2 ê=(e, ..... ek_)E(T) - 
1 .-.k_l 
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The latter formula can be restated as 

(3.4) 

 kï  ¢,1¢,2 ' ¢«.k = 0. 
k=l ê=(el ..... et_)EE(T) - 
e --" -ek-  

(Here we must interpret the k = 1 summand as (T.) 
Pro@ Let us define bT by (3.3) and set a' 
A' = ETeT dT)T" Çe have 
xp(A'). =  . 
k=O 
=   ç 
k=O 
1 
= (T ..... T)e 
1 
Te 

= ETT(T)T, Ai = a'D. Then 

(T1 ..... Tk)(T  

Using the fact that )T  z = 

T(T k--12 (T ..... T)(T  

[)T1 )Tll "'" [)Tk-l)T/-l] [¢Tk 

Applying Corollarv 2.10, substituting t "(i) = çbT," 

TT k=2 T:£ ê=(el ,...,e_):E(T) - 
v(T)_>2 e >---->-e_  

dT¢, 1 " " " dT¢,k ) T  

Letting S' be the singleton tree: 

TT ê-----(e ):E(T) t- 
v(T)_>2 ..... e-i 
el -.-. -e/¢_ 1 

Since IPs = H, and, by definition, bs = 1 and the sum in parentheses is 0: 
=z+H=F. 
By the uniqueness property of a we must have a  = a. 
expression (3.2) for a follows froln Theorem 2.4. 

The Ulfiqueness of the 
[] 

Chains and Shrubs. Two special types of trees are the "chains" and the "shrubs", 
mentioned in §2. Given an integer n >_ 1 we let C  OE be the chain with n 
vertices, which is the unique rooted tree in E of height n - 1. For n > 0 we let 
S  E+ be the shrub with  + 1 vertices, which is the Ulfique rooted tree in E+ 
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of height < 1 (equality holds lmless n = 0). Note that C I = S 0 = o, the singleton 
t, ree. 
By using the recurrence forlmlla (a.a), we e,. calculate ¢ for chains and shrubs 
 follows. Çonsider the generating functions 
Oe X  
() =  ¢ 

Theu we have: 

Corollary 3.5. 
(3.5) 
(3.6) 
In partzcular. 

The generating functions c(x) and s(x) are given by 
(a) c(x) = 1,1(1 + x), 
(b) s(x)- e - 1" 
we bave ¢c. = (-1) n-il for ail n > 1 and Cs = b, where 

 X n 
b0, bi, b2, .. . are the Bernoulli numbers 3 defined by «z_ -- Yn=0 b.,. 

 etc, Cr,="" Cr, k ) 
ê=( e ..... ek_  ) E( Cn )-  
el --'ek- 1 

Proof. (a) According to (3.3) we have 
rte---2 

Xrto 

Noting that v(C,) =  and each Tg, j is also a path: 

n=2 k:2 (i,...,it)El  j=l 
i÷-..÷i=n 

: x- E (coefficient of m in c(x) ) x' 
n=2 k=2 
= x -  coefficient of x  in c(x) x' 
n=2 k=2 
=z- coecientofz nin 
n=2 k=2 " 
=  - (e c() - c(x) - 1). 

3This indexing and signage differs from an alternate definition of the Bernoulli numbers as the 
sequence B, B2,.-. defined by 
1  Bn x2 n 
x -- 1 - - + E (-1)'z-' (2n)! " 
e x - 1 2 
Thus the relationship is Bn = (-1)n-b2n for n _> 1. 
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Noting that v(S,) = n + 1 and precisely one Tê, j is a shrub with ail others being 
singletons: 

(3.7) 

= l - x -ls(x)(e x-x-l). 

(3.8) 
Solving for s(x) in the equation s(x) = 1 - x-ls(x)(e x - x - 1) gives (3.6). [] 
3.3. Polynomial Coefiïcients ¢T(t) of Formal Flow. Let us first recall the 
formal flow Ff = exp(tA)  z and some of its properties. See [Z] for more details. 
Definition 3.6. Given an indeterminate t, define the systenl Ff  C[t][[z]] n by 
(3.9) Ff = exp(tA)  z. 
It is called the formal flow generated by F. 
It is easy to verify that Ft  C[t][[z]] '. Therefore, a specialization t = , for 
any   C (or  in any C-algebra), makes sense. According to Proposition (3.1), 
setting t = 1 in Ff recovers F. 
The following proposition shows that t behaves as an exponent for F. 
Proposition 3.7. Let t and s be indeterminates. Then 
Fs+t = Ft o Fs. 
Hence setting t = n in Ft, for n  N, gives the n-fold composition F o ... o F, and 
setting t = -n gives the n-fold composition F -1 o ... o F - of the formal inverse. 
In particular, 
Ftl,=_l = F-l" 

Proof. We bave 

F,+t = exp((s + t)A). z = exp(sA + tA). z 
= exp(sA)- exp(tA)- z = exp(sA)- 
We use the fact that exp(sA) is a C-algebra automorphism of C[s, t][[z]] (see Remark 
3.2): 

= Ft(exp(sA. z)) = Et(F,) 
= FtoG. 
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Thus Ff can be viewed as the "firmal t-th power of F". 
The system Ff can be expressed in terres of the tree expressions TT as follows: 

Theorem 3.8. There exists a untque ,set of polgnomials {/,T(t)  Q[t]} indexed by 
the set of rooted trees T  "IF such tbat 
(3.1o)  = z +  ¢(t). 
These polynomials are given by the formula 

(3.11) 
k=l ê=(el ..... ek-I)EE(T) 
e i ,-... ,- ek_ 1 
( Again 

Now we apply Corollary 2.10 to the k >_ 2 summands: 

=z+ 

 t k 
ZZ Z 
k=l T" ê=(ea,...,ek_a)E(T) 

=z+ }2 

CTg,1 " .. CTê, k ) T 

, CT«,I ... CT«,) 
ê=(ex ..... ek_a)EE(T) -a 
el -----ek_ 1 

This gives the desired result. The uniqueness of DT follows from applying stable 
linear independence (Corollary 2.6) to each power of t in (3.10). [] 

Lemma 3.9. For any T  "l, we have 
(1) if T is the singleton, we bave 
(2) DT(0) = 0. 
{1 if v(T) = 1, 
(3) çT(1)= 0 if t,(T) _> 2. 
(4) ¢-(0) = 
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Pro@ Ail statements above follow immediately from (3.11), except the assertion 
T(1) = 0 when v(T) _> 2, which is exact.ly (3.4). [] 

Forests. The fornmla (3.11) defines a torique polynomial YT(t) for each rooted 
tree T. A forest is the disjoint union of finitely many rooted t.rees. We extend the 
definitions of (p and {,p(t) to any forest P as follows: 

Definition 3.10. For any forest P that is the disjoint union of rooted trees 
Tt .... ,Tk, we define Ce to be ¢T1 if k = 1 mld 0 otherwise. Define ¢p(t) = 
k 
H,=I er, (t). 

Lemma 3.11. Let T be a rooted tree with, v(T) >_ 2. For any proper rooted subtree 
T' of T we bave 

(3.12) 

er\r,(0 = y.., 

k=l 

ê=(e ..... e)E(T)  
el »-... »-e k 
T,+I =T' 

Pwof. Let T [il (j = 1,2 ..... d) be the commcted components of T\T', and let 
0 be the edge of T that connects Tb] with T'. Note that from fixed sequences 
ej 
ej,1 - ej,2 - ... >- ej,k  E(T IiI) with bi + ku + --- + kg = k -- d, appended bv 
the edges e, we can get ((1  ! 
+l),(k2+l),...,(kd+l))  (k+l)!(k2+l)!.-.(kd+l)! different 
sequences ci - e2 - --- - ek  E(T) such that Tk+l = T . Therefore, 

v(T)-  t 
k=l (k ,...,kd)_N d 
k+k=+...+ka=k-d 

d 

CTê.1 CTê,2 "'" CTê, k 

k 

(k, + 1)!(k + 1)!---(kg + 1)! 

CT),I CTj,2 "'" CTj.kj+l 

The last equality follows from (3.11). 

The lemma above allows us to prove the following theorem. If we let 0 be the 
empty tree and define {Po = z, then Theorem 3.8 can be seen as the special case 
S = 0 of the t.heorem below. 
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Theorem 3.12. For any rooted tree S, we have 

Proof. 

 t A   Ts 
exp(tA)- TS = E E 

=  V.,. ¢ « 
k=O 

 t k 
=+ . , 
k=l (T ..... T)eT  

[er, ).r,]--- leçon)r,]  Ys. 

Apply Corollary 2.9, substituting 
 

 t k 
k=l TT 
v(T)>l 

E 
TT 
v(T)_>l 

E 
ê=(el ..... ek-I)6E(T)  

k=l 

¢T«. "'" ¢ç«, TF 

ê----(e .... ,ek)6E(T)  
el.--ek 
T«, +  --S 

T [T'<_T 
\T'_S 
The lst equality follows fro*** Lemma 3.11. 

Proposition 3.13. For any rooted tree T, we bave 

() 
(3.14) 

(b) 
(3.15) 

or, in other words, 

(3.16) 

e6E(T) 

¢() = ¢ +  
SeT 

¢(t) 

= ¢(0)+  ¢o,,(0)¢o.(t) 
eE(T} 
= ¢(0) +  ¢(0)¢\s(t). 
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0 

Applying Corollary 2.10 with k --- 2, setting }1) = dT, ),/2) ____- 'T(t) for ail T E "F: 

But we also have, by Theorem 3.8, 

0 
(3.17) c9- Ft = E b(t)[PT. 

Comparing the coefficient of [PT, and appealing to stable linear independence 
specifically, Corollary 2.6--we get (3.14). (We use the fact that polynomial fimc- 
tions that agree at ail a E C must be equal.) 
(b) 

0 0 
c- Ft = - exp ( t A ) . z 
-- A-exp(tA)  z 
=- exp(tA)  A- z 
= exp(tA)- a. 

Applying Theorem 3.4: 

= exp(tA)- E 
SET 
---- E çb8 exp(tA)- 
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Applying Theorem 3.12: 
TT TT 
Comparing this with (a.lî), and again employing Corollary 2.6, we get (a.15). 

An interesting consequence of the proposition above is the following recurrence 
formula for eT in terres of the number of the leaves of T. 

Proposition 3.14. For any rooted tree T, suppose that the foot rtT bas d children. 

I(T) 
(3.8) 

__, eT\{vl,v2 ..... v} - Sd, l)T\{rtT}. 
{ri ,2 ..... v}C_L(T) 
?-'1 ,Va .... ,V r distinct 
Proof. From (3.14), setting t = 1, we get 
(3.19) bT(l) - [ eT -J- eT\{rtT} if d = 1, 
eT if d>2 
since, by Lemma 3.9, ¢T2(1) = 0 except when T2 is the singleton. 

Prom (3.15), 

setting t = 1, we get 

l(T) 
(3.20) bT(1) = eT + Z E eT\{vl,v2 ..... v} 
r=l {vl ,v= ..... v}C_L(T) 
ri ,v2 .... ,v distinct 

since ¢T\S (1) = 0 except when T\S is the disjoint llilioll of finitely manv singletons. 
Crnparing (3.19) and (3.20) gives (3.18). [] 

Before leaving this subsection, we will do some calculations on the polynomials 
eT(t) for the chains C, and shrubs 
Consider the generating functions e(t,.) = =0 ¢c (t)«  (t ¢Co(t) = 1) ad 
S(t,.) = =0 

Corollary 3.15. 
() 

(3.21) 

or. in other words. 

(3.22) 

The generating functions g(t. x) and $(t, x) are given by 

e(t,,) = exp (t In(1 + x)) = (1 + x) t 

¢c.t)=() =t(t-1)'"(t-+l)--7,! 
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e t - 1 
(3.23) S(t, x) -- 
e x- 1 

Pro@ (a) By Theorem 3.8 and Corollary 3.5 we have 

=I+E 
(m ,m2 ..... k)E('+)  
"rrtl ÷m2÷---÷mk -n 

 vil 
= et(-z+- - .... +-*'"+... ) 
= exp (tln(1 + :)). 
(b) Similarly, we have 
oe v( S,J a 
n=O k=l 

¢Ta ¢T.a "'" ¢T, x 

(-1)m (-1) m= (-1) m 
ml m ma 

 CTo,¢To.= '" CT,k 
ê=(e ..... e_l)E(Sn) - 
el -.'ek_ 1 

m X n 

Noting that ail but one of CT,: .... , CT,, are singletons, the remailfing one being 
Sn-k+ l : 

Replacing n by n - 1: 

 k (t)a 
: X--1 E ]î,-Ç--.bn_ a 
n:l k=l 
= x-(et - 1)e: - 1 

e t - 1 
(3.24) -- 
e : - 1 

(n - k)! 

Remark 3.16. The formulas of Corollary 3.15 can also be easily derived from Theo- 
rem 4.2 in the next section. But we think the calculations above are more intriguing. 

4. THE hlAIN THEOREM 

4.1. Main Theorem on bT(t). In the last section, we defined the polynomial 
T(t), for each rooted tree T (see Theorem 3.8). For each rooted forest P, i.e., the 
disjoint union of finitely many rooted trees Ti (i = 1,2,..., k), we also defined bp 



3136 DAVID WRIGHT AND WENHUA ZHAO 

(see Definition 3.10). Recalling from §2.1 the definition of a rooted subtree, we are 
now rea(h" t.o l)rove the fillowing main theorem. 
Theorein 4.1. Let t and s be indetermin«tes. For T  T we have 
,,here the last sure ru,s over dl pvp«r moted subtrees T' of T. 
Pro@ Clearh" ,'xp((t + s)l), z = exp (t.4)- exp (sA)-z. So we ha 
: +  'r(t + s)r = exp(( + s)A)-z = exp (L4)- exp (sA)  z 
T 
= z +  'r(¢)r +  g,r(s)(exp(tA) - 

At)plying Theorem 3.12 to exp(tA)- 

= Z + E dT(t)PT+ E dT(S) ( pT + E ( E dS\S'(t))Te3r T3r 563r 
\S'T 

TT S' <S 
S' --T 

T6T T' <T 

S 

According to Corollarv 2.6 and an easy specialization argument, the theorem follows 
bv comparing the coefficients of T in the above equation. [] 
The difference polynornial of g(l)  C[T] is defined to be the polynomial Ag(t) = 
g(t+ 1)-g(t). The following special case of the I heorem above, which gives a fornmla 
for the difference polynomial of T(t), is most useful to us. 
Theorem 4.2. For any tree T with v(T) >_ 2, we bave 
,l'T(t ) = dTx (IQ)T(t) " " " dTa(t) 
-- 
,,,here T,. i = 1.2 ..... d are th« «om, e«t«d «omponents of T\ { l't T }. 



D-LOG AND FORMAL FLOW FOR ANALYTIC ISOMORPHISMS 3137 

Pro@ This follows form Theorem 4.1 by setting s = 1 in (4.1) and appealing to 
Lemnm 3.9, which says that PT(1) = 0 unless T is the singleton, in which case 
hT(1) = 1. [] 

Theorem 4.3. For any tree T with v(T) >_ 2, we have 

l(T) 

(4.2) a+T(O = F.  
r=l {v],v2 ..... v,.}ÇL(T) 
vl ,v2,...,v distinct 

(b) 

l(T) 
(4.3) ,hT\trt-t (t) = E E PT\t,,,2 ..... ,}(t) 
r=l {v,v2 ..... vr}ÇL(T) 
v ,v .... ,v distinct 
where T, i = 1,2 ..... d are the connected components of T\ {rtT}. 
Pro@ Clearly, (b) follows ff'oto (a) and Theorem 4.2. For (a), switch t and s and 
set s = 1 in 4.1 to get 
PT(t + 1) = PT(t) + eT(l) +  CT\T,(1)PT'(t). 
T' <T 
By Lemma 3.9, we have hT(1) = 0 and ,hT\T,(1) = 0, unless T\T' is a disjoint. 
union of singletons, in which case ,hT\T, (1) = 1. Therefore, 

liT) 
hT(t + 1)- hT(t) = E 

Z 
{v .v2 ..... v } ç L(T) 
vl ,v2 ,...,v distinct 

as desired. [] 

4.2. Algorithm for WT(t). From Theorem 4.2 we get the following algorithm for 
computing T(t). Here, for h(t)  C[t], Ah(t) is defiued to be the unique poly- 
nomial g(t)  C[t] such that Ag(t) = h(t) and g(0) = 0. 
Algorithm. For any fixed rooted tree T, we sign a polynomial N(t) to each vertex 
v of T as follows: 
(1) For each leaf v of T, set N(t) = t. 
(2) For any other vertex v of T, define N (t) inductively starting from the high- 
est level by N (t) =/k- 1 (Nv (t)N2 (t)... N, (t)), where vj, j = 1,2 .... , k, 
are the distinct children of v. 
Then for each vertex v of T, N(t) = pT,+,(t), where T + is the subtree of T rooted 
at the vertex v. In particular, we have çT(t) -= NrtT(t). [] 
The following exmnple applies this algorithm to the shrubs S to show that the 
polynomials Cs (t) are closely related to the Bernoulli polynomials Bn(t) defined 
by  = -_oB(t)-.. (Compare this with (b) of Corollary 3.15.) 
Example 4.4. Let Vl,...,v be the leaves of the shrub S. Following the al- 
gorithm, we first assign the polynomial t to each leaf vi. The next step in the 
algorithm gives 
(4.4) hs (t) = A-(t). 
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One of the fimdamental properties of the Bornoulli polynomials B,(t) is 
(4.5) (t) = ,(t + l)- (t)= t - , 
and ri'oto this and the fact that  commutes with  one eily derives 
d 
(4.6) B.+l(t) = (n+ 1)B(t). 
From (4.5) and (4.6) we get 
t B.+I (t) - B.+I(0) 
(4.7) &-l(t') = B,,()«,, = 
n+ 1 
Putting togcther cquations (4.4) and (4.7), we obtain this relationship between 
g's, (t) and B,+l (t), 
¢s,(t) = B,,.(,)du = B+i(t)- B.+,(0) 
n+ l 
4.3. Combinatorial Property of T(t). After the main part of this work w 
donc, Professor .lolm Sharcshian pointed out to us that the polynomial 'T(t) for 
rootcd trees coincides with the strict ordcr polynomial (P,t) for finite posets 
(partial ordered sers) P in «ombinatorics (sec Chaptcrs 3 and 4 in [Stl]). X first 
re«all the polynomial (t(P, t) ssociat.ed with a fiuite poser, and then we show that, 
when P is the poser of the set V(T) of vertices of a rooted tree T with the natural 
partial order iuduced by an«estry (the root being the unique smallest element), we 
hae ",(t) = ff(P, t). 
Any rooted tree corresponds in this way to a unique finite poset, and a finite 
poser P corresponds fo a rooted tree pre«isely wheu it satisfies these two criteria: 
(1) P has a unique smallest elemcnt, and 
(2) any interval in P is totally ordered. 
For any n G N, the chain C gives thc totally ordered poser with n elements. (X 
tan view it as the set {1, 2 ..... t} with the natural order of the positive integers.) 
For any poset P, we say that a map f  P  C is stct order-presemin 9 if 
f(a) < f(b) in C whenever a < b in P. It is well known that there exists a unique 
polynomial ()(P, t) su«h that Ç(P, n) equals the number of strict order-preserving 
maps f from P to C for ail n G N. This, then, is the theorem shown to us by John 
Shareshian. 

Theorem 4.5. For any rooted tree T, we have 
(4.8) Z['T(t) = ((T,t) 
(where. on the right. T is viewed as a finite paset as deseribed above). 
Proof. It is obvious that when T is the singleton, r(T, t) = t. Hence it is enough 
to show that r(T,t) also satisfies the recursion formula of Theorem 4.2. More 
precisely, we will show that, in the notation of Theorem 4.2, we bave 
Ar(T,n) = T, n + 1) - r(T,,) = (Tl,n)r(T.2, t)-.- r(Ta, n) 
for any n G N. 
Note that Af(T, n) equals the number of strict order-preserving maps f from 
T to C,+a = {1,2, .... n + 1} such that f(rtT) = 1. But this number is also the 
saine as the number of strict order-preserving maps g from T\{rtT} to C,, which 
is r(T1, n)r(T2, .)--. fi(Ta,-). [] 
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Remark 4.6. It is interesting that the strict order polynonlial ((T, t) for the finite 
posets induced by rooted trees T cau be defined in a totallv different way, nmnely, 
accordiug to the fornmla (3.11) of Theorem 3.8. Iu fact, this realization of the 
strict order polyuonfial can be generalized to an arbitrary finite poset P. This 
generalizatiou and its consequences will be discussed in the upcoming paper [S\VZ]. 

5. SOME APPLICATIONS 
For a fornlal autonlorphism F = (F1, F2, .... Fn) = z + H of the form identity 
plus higher, we give a restatement and ll(,W proof of the tree fornmla for the fornml 
inverse rirst proved in [BC\V] and [W21. 
Theorem 5.1. For any roo{ed tree T, we hae *Pr(-1) = (-1) {r). Hece {he 
formal im,erse F -1 of F is 9iven by a 
(5.1) F-' = Z -}- Z(--1)v(T)T . 
Proof. The formula (5.1) follows fronl bT(--1) = (--1) v(T) by Proposition 3.7" aud 
Thcorem 3.8. 
It is well kuown iii COllbilmtorics (see [SII]) that the strict order polynoulials 
satisfy çt(T,-1) = (-1) v(y), froIn which the result follows, iii light of Theorem 4.5. 
For conlpleteuess, we give a direct proof here. 
\Ve use t.he umthematical iuduction on v(T). The case for v(T) = 1 is trivial. 
(bT(t) = t iu this case.) Sui)pose t,(T) k 2. Bv Theorem 4.2, setting t = -1, we 
have 

bT(0) -- çT(--1) ----- çT1 (--1)çT2 (--1) "" "çTa(-1) 
where Ti, i = 1,2 ..... d are the connected components of T\{rt.T}. We have 
çT(0) = 0, and bv induction we may assume the theoreln holds for Tl,...,;Fa. 
Hence 
bT(--1) = --(--1) vr` +vra+...+vr. = (_l)V(T). 

It is known that the Jacobian coujecture (see [BCW] for a statemeut of this 
fmnous problem) is equivalent to the a,sertion that. 

(5.2) Z T = 0 

T6"lï2 

for N » 0 whenever H is a honlogeneous polynomial system (of degree >_ 2) and 
the Jacobian determinant I(DFi)I is (everywhere) nonzero. In fact, this follows 
from Theorem 5.1, since when H is homogeneous the polynonfials YTe' PT, for 
fixed N, are the homogeneous summauds of F - (see 1Remark 2.2). When H is 
holnogeneous, the condition I(D F)I = 1 is known to be equivalent to the nilpotence 
of the Jacobian matrix JH = (DH) (see [BC]). Thus the following result 
pïesents an intïiguing statement for compaïison. 

4The formula as given in [BCW] and [W2] did not include the factor (-1) v(T). It appears here 
because of our choice in writing F = z + H instead of F = z - H. 
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Proposition 5.2. Assume H is homogeneous of degree >_ 2. For any rooted tree, 
let hT, k be the number of vertices of height k. Suppose that (JH)  = O. Then 

(5.3) Z hT, mT - 0 

for any N E bl and m >_ k. 

Pro@ Suppose that deg H = d _> 2. It follows from Euler's formula that JH-(z t) = 
(dH) t, from which we get ?i(JH) k. (2 t) = (JH) k-i -H t = 0. (Here the superscript t 
denotes trmlspose, converting a row to a cohmm so that the matrix multiplications 
make sense.) For any intcger m _> 1, a straightforward calculation shows that the 
chain C, has thc property Te,, -- JH m-  H t  Therefore, Te,,, = 0 for rn _> k, 
and we have 

0 = exp(-A)- 

By Theorem 3.12, setting S = (' and t = -1 in (3.13)" 

By Theorem 5.1: 

In particular, for any N E N, we bave 
 (-1)-m = (_1)- 
which gives (5.3). 

Z hT, mT 

=0. 

The proposition above shows that, for a fixed honogeneous polynomial system 
H. the polynomials PT are in some sense quite linearly dependent on each other. 
Finally, let us point out that the formal flow Ff gives a formal flow between F 
and the identity map id, i.e., Ftl,=, = F and Ftl,=o = id, having the additional 
properties Ff(O) = 0 and JFt(O) = I.,. It is an open question in complex analysis 
whether, for any local analytic map F, such an analytic flow exists. The usual 
approach to this question is to show that F is linearizable, i.e., it is conjugate to 
a linear map. But when F is linearizable the question is still open, even for the 
one-variable case. (There are many partial results for this problem.) So it is of 
interest that the formal solution to this question is given by the very clean fornmla 
(3.10) of Theorem 3.8. But the question of when F is locally convergent is still 
open. 
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A GENERALIZATION OF TIGHT CLOSURE 
AND MULTIPLIER IDEALS 

NOBUO HARA AND KEN-ICHI YOSHIDA 

ABSTRACT. We introduce a new variant of tight closure associated fo any fixed 
idem a, which we call a-tight closure, and study various properties thereof. In 
our theory, the annihilator idem T(a) of ail a-tight closure relations, which 
is a generalization of the test idem in the usual tight closure theory, plays a 
particularly important role. "We prove the correspondence of the ideal T(a) and 
the nmltiplier ideal associated to a (or, the adjoint of a in Lipman's sense) in 
normal Q-Gorenstein rings reduced from characteristic zero to characteristic 
p » 0. Also, in fixed prime characteristic, we establish sonm properties of 
T(a) similar to those of multiplier ideals (e.g., a Briançon-Skoda-type theorem, 
subadditivity, etc.) with considerably simple proofs, and study the relationship 
between the ideal 7-(a) and the F-rationality of Rees algehras. 

INTRODUCTION 

The notion of tight closure, introduced by Hochster and Hmeke [HH1] lnore 
than a decade ago, has emerged as a powerfld new tool in comilmtative algebra. 
Tight closure gives remarkably simple characteristic p proofs of several results that 
were not thought to be particularly related, e.g., that rings of invariants of linearly 
reductive groups acting on regular rings are Cohen-Macaulay. that the integral 
closure of the th power of an n-generator idem of a regular ring is contained in the 
ideal (the Briançon-Skoda theoretn), and so on. Also, the notions of F-regular and 
F-rational rings are defined via tight closure, and they turned out to correspond 
to log terminal and rational singularities, respectively ([Hall, [HW], [MS], [Sml]). 
This result is generalized to the correspondence of test ideals and multiplier ideals 
([Ha2], [Sm2]), both of which play very ilnportant roles in the tight closure theory 
and birational algebraic geometry, respectively. 
The test ideal of a ring R of characteristic p, dcnoted by -(R), is the almihilator 
ideal of all tight closure relations of R. On the other hand, the notion of multiplier 
ideals has several variants. Originally, a multiplier ideal was defined analytically 
for a given plurisubharlnonic function on a COlnplex almlytic lnanifold [N]. This 
is reforlnulated in the algebro-geometric setting (in characteristic zero) in terres of 
resolution of singularities and discrepancy divisors ([Ei], [La]). Actually, two types 
of multiplier ideals are defined in this setting, that is, the multiplier ideal dZ(D) 
associated to a Q-divisor D and the nmltiplier ideal dZ(a) associated to an ideal a. 
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The latter is also defined by Lipman [Li] in a more algebraic context and is called 
the "adjoint ideal". 
Precisely speaking, the multiplier ideal that is proved to correspond to the test 
ideal r(R) is the one associated to the trivial divisor D = 0 or the unit ideal 
a = R, which defines the non-log-termina] locus of SpecR. In most applications, 
however, the nsefulness of multiplier ideals is obtained by considering multiplier 
ideals associated to various divisors or ideals; sec, e.g., [Ei], [La], [Li]. Thus we are 
tempted to define a sort of tight closure operation and a test ideal'" associated to 
any given Q-divisor or ideal. 
In this pal)er, we introduce a generalization of tight closure, which we call a-tight 
closure, associated to an ideal a, study various properties (including the relationship 
with multiplier ideals), and give some applications of a-tight closure. (In [T], the 
reader can find an attempt to genera]ize tight closure in the other direction, that 
is, Atight ch»sure associated to a Q-divisor A: sec also [tlW].) Actuallv, given an 
ideal a of a Noetherian ring R of characteristic p > 0, we define the a-tight closure 
I *a of an ideal I C R to be the idem consisting of all elements z  R for which 
there exists an element c  R hot in any minimal prime ideal such that 
cz p a p C I [p] 
for ail e » 0, whcre [[P] is the ideal gcncrated bv thc pth povers of elements of 
I; sec Definition 1.1 and also Dcfinition 6.1 for filrthcr generalization to "rational 
coetïficients "'. We then define the ideal 7-(a) of R to be the unique largest ideal such 
that 7-(a)I * C I for ail ideals I C_ R. St». in the case where a = R is the unit ideM, 
the a-tight closure I *« = I *R is equal to the tight closure I* in the usual sense, and 
the ideal "r(a) = 7-(R) is nothing but the test ideal [HH1]. 
There are many similarities between the usual tight closure and a-tight closure. 
For example, the existence of n-test elements (Definition 1.6) is proved quite sim- 
ilarly as in the case for the usual test elements. But there does exist a difference 
as well: \Ve require a "closure" opcration to satisfv the property that, once the 
operation is performcd, the obtained closure does hot change if one performs the 
operation twice or more, and tight closure satisfies this property, namely, (I*)* = I*. 
However, it happons that (I*«)* is strictly larger than I*, and so, a-tight closure 
is hot in fact a closure operatiou. Sinfilarly, unlike the usual test ideal "r(R), the 
ideal 7-(a) is no longer equal to the one gencratcd by n-test elements if a Ç R. 
In spite of the apparent disadvantage mentioned above, we find many more ad- 
vantages in the circle of ideas involving a-tight closure. The significance of the ideal 
7-(a) is witnessed by the following theorem, which ensures the expected correspon- 
dence of 7-(a) and the multiplier ideal ff(a); sec also Theorem 6.7. 
Theorenl 3.4. Let R be a normal Q-Gorenstein local ring essentially of finite 
type over a field, and let a be a nonzero ideal. Assume that a C_ R is reduced 
from characteristic zero to characteristic p » O. to9ether with a lo 9 resolution of 
sin9ularities f" X -- Y = Spec R such that a(.3x = Ox(-Z) is invertible. Then 
r(a) = H°(X, Ox([Kx - f*tçy] - Z)). 
Note that, by definition, the multiplier ideal ff(n) in characteristic zero takes 
just the saine form as the right-hand side of the above equality. So one can think of 
the right-hand side as a reduction modulo p of the multiplier idem in characteristic 
zero. This theorem generalizes the main results of [Ha2] and [Sm2], and is proved 
essentially in a similar way as in those papers, with a little more effort. 
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The useflfiness of nmltiplier ideals is tied up with vanishing theorems in charac- 
teristic zero such as thc Nadcl vanishing theoreln [NI, which rail in characteristic 
p > 0. For exaInple, Lipman used these tools to establish au improved version of the 
Briançon-Skoda theorem, which asserts that, for any ideal a of a regular local ring 
generated by r elements, one has ff(a +T-1) C a  for ail n _> 0 [Li]; sec also [La]. 
We will prove in Theorem 2.1 that the corresponding statement "r(a +T-) C_ a  
holds true in characteristic p > 0. Taking into account the correspondence in The- 
orem 3.4, we see that this gives auother proof of Lipman's result.. An advantage of 
our prime characteristic proof here is that it is quite elementary and simple (like 
the original tight closure proof of Briançon Skoda [HH1]) and does hot depend on 
desingularization or Valfishing theorelns. 
We shall take a look at the organization of this paper, which we hope gives 
further coIfirmation of the usefuluess of a-tight closure. 
After discussing basic properties of a-tight closure aud the ideal T(a) in Section 1, 
we give three fundaluental applications of a-t.ight closure in Section 2. The first 
one is the modified Briançon-Skoda theorem via a-tight closure meutioned above. 
Second, we study the relatiolship between a-tight closure and tight integral closure 
defined by Hochster [Ho2], and rephrase the F-ratiouality criterion of Rees algebras 
obtained iii [HWY1] in terlns of a-tight closure; sec Theorem 2.7 and Corollary 
2.9. This enables us to give an interesting characterization of regular local riugs 
as the third application. Nmnely, we prove in Theorem 2.15 that the regularity of 
a d-dimensional local riug (R, rn) is characterized by the property that T(rn d-l) = 
R. This is considered an aualog of the fact that the weak F-regularity of R is 
characterized by the property that T(R) = R. 
Section 3 is devoted to proving the equality r(a) = 7() in Theorem 3.4, v«hich 
holds truc in the situation reduced from characteristic zero to characteristic p » 0. 
We note that the contailunent -(a) C ,(a) essentially holds truc iu any fixed 
characteristic p > 0; cf. Proposition 3.8. 
In Section 4, we establish properties of ideals -() in fixed characteristic p > 0 
similar to those of multiplier ideals 7() in characteristic zero, namely, the restric- 
tion theorem (Theorem 4.1), the subadditivity (Theorem 4.5), and a description of 
ideals r(a)in the toric case (Theorem 4.8); sec [DEL], [How], [La] for the results 
proved for multiplier ideals. Again in light of Theorem 3.4, we tan also say that 
the results in this section give uew prime characteristic proofs of the geolnetric 
statements for multiplier ideals in characteristic zero, although we work with the 
Probenius map in fixed characteristic p > 0. 
In Section 5, we explore the behavior of the ideal r(I) for an m-primary ideal I 
of a Gorenstein local domain (R, m) of characteristic p > 0, from the viewpoint of 
the Rees algebra R(I) = R[It] via Theorem 2.7. As a main result of this section, we 
prove in Theorem 5.1 that if R(I) is F-rational, then it.s graded canonical module 
is described as l:t() = (>1 -(I) U" In particular, the equality r(I) = 7(I) 
holds if R(I) is F-rational, and the converse is also truc in t.he case of a two- 
dimensional rational double point. (It should be noted that the ideals "r(I) and 
7(I) may disagree in fixed positive characteristic.) Comparing Theorem 5.1 with 
Hyry's results ([Hyl], [Hy2]), we tan also deduce various results for "r(I). 
In Section 6, we extend the notions of a-tight closure and the ideal "r() to those 
with "rational coefficients", and generalize some results discussed in the previous 
sections to the case of rational coefficients. Although we bave no explicit applica- 
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tions of this generalization at the lnonlellt, we include this sectiou for future ref- 
erence, l»eca.use recent applications of multiplier ideals involve rational coefficiems 
SllCCeSSflllIy ([ES], [Lai). 
Before going ahead, we review part of the notation and basic notions of the 
tight closure teory. We keep it lninimal to avoid overlap with Section 1. The 
read«r is referred to H,,«hster and Hun«ke [HHI]-[HH3] and Hun«ke [Hu] for the 
fifll development of the theory. 
Notation and basic notions. Throughout this paper ail rings are Noetherian 
commutative rings with unity. For a ring R, we dcnote bv R  the set of elements 
of R that are hot in any nfininml prime ideal. V will oftel work over a field of 
«haracteristic p > [). In this case we alwas's use the letter q for a power pe of p. 
Let R be a Noctherian ring of characteristic p > 0. r an ideal I of R and a 
power q of p, we denotc hy I Iv] thc idcal gencrated by the qth powers of elelnents 
of I. The tight closure I* of I is defined t.o l)c tbe ideal consisting of ail elelnelltS 
z  R %r which there exists an element c  R  such that cz q  I [ql for all lin'ge 
q = pe. (Tight closure is also defilled for a submodule of a. module; cf. Section 1.) 
We sav that R is w«akl# F-r«gMar if every ideal I of R is tightly closed, that is, 
I* = I. A local riug (R.m) is said to be F-rationol if every kleal generated by a 
SyStClU of parmneters of R is tightly closed. In general, we say that R is F-rational 
(resp. F-regular) if ail of its local rings are F-ratiollal (resp. weakly F-regular). 
Let F: R  R be the Frobcnius lnap, that is, the ring homomorphisln sending 
z  R to z p  R. Thc ring R vicwed as an R-module via the e-tilnes iterated 
Frobenius map Fe: R  R is denotcd by eR. XX say that R is F-finite if R is a 
finitely generaed R-module. If R is reduced, theu Fe: R  eR is identified with 
the natural inclusion map R  R /q. An F-finite reduced ring R is said to be 
.strongl F-regular if for every elelnent c  R °, there exists a. power q = pe such that 
the inchlsion map c/qR  R /q si)lits as an R-module homoluorphism. 

1. DEFINITION AND BASIC PROPERTIES OF a-TIGHT CLOSURE 
Let / be a Noethcrian ring of characteristic p > 0 and let 3I be an /-lnodule. 
For each e  N, we define lFe(ll) = IFÏ(/[ ) := / OR M and regard it as an R- 
nlodule by the action of R = eR ff'oto the left. Thon we have the induced e-tinles 
iterated Frobenius nmp F " M --, IFe(5l). The image of z  M via this nlap is 
denoted bv z q := Fe(z)  IF«(M). For a.n/ï}-sublnodule N of iii, we denote bv V[q] 
the image of the iuduced lnap IFS(N) -- 
Definition 1.1. Let a 1)e ail ideal of a Noetherian rillg /ï' of characteristic p > 0 
aud let. N C_ 5I be R-modules. The a-tight closure of N iii 1I, denoted bv Nî, is 
defined to be the submodule of BI consisting of ail elements z  BI for which there 
exists c  R ° such that 
czq(1 q ç Nï I 
for all large q = p. The a-tight closure of an ideal I C R is just defined by 
I*" = 
Remark 1.2. (1) hl the case where a = R is the unit ideal, the a-tight closure 
N;î = N;î is nothing but the (usual) tight closure N, as defined iii [HH1]. 
However, unlike the UStlal tight closure, it ma:g. happen that t-'tV*«*«M M is strictlv 
larger than Nî; see Remark 1.4 (1). In this sense a-tight closure is hot an "'honest'" 
closure operation iii general. 
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(2) It seems significant to extend the definition to "rational coeflCicients '', if we 
take into a«count the relationship with multiplier ideals: sec [Lai, [T] and Sections 
3 and 4. Namely, given nomlegative t  Q and  Ç R, N ç M as in Definition 1.1, 
we can define file t- «-tight Ho,sure nr.t. of N in I and generalize some results for 
-tight closure to those for t  -tight closure. b treat this issue in Section 6. 

We collect Solne basic properties of -tight closure in the following. The proofs 
are easv and are left to the reader. 

Proposition 1.3. Let  and b d«note id«als of a Noetherian ring R of characteristic 
p > 0 and let L and N dcnote submodules of an R-module M. 
() N 
= OtlN. 
(2) If L 
(3) 5;* ç (Nî  b)at. Moreover, if b is a principal ideal, then the equality 
t = (t " b)t holds. 
of c. the the equati@ Nff = NÏ holds. 
Remrk lA. (1) I folh,ws that zt ç Aî Ç (N;t  «)t from (3) and (4) of 
Proposition 1.3. If  is a principal ideal, lhen the equality on the right occurs, and 
(M)M = (M " 2)M is strictlv, larger than N M = (Gt " a)al in general. 
(2) The colon-capturing property [HH1, Section 7] says that parameters behave 
like a regular sequence modulo tight closure. Namely, if xl,...,Xi+l G R are 
parameters, then under a mild assumption, (xa,...,xi) :n xi+ Ç (x,...,xi)*. 
Since a-tight closure contains the usual tight closure, this remains truc if we replace 
tle usual tight closure by a-tight «losure. In Proposition 1.5 below we slightly 
improve this colon-capturing property for a-tight closure using the existence of a 
test element. Sec [HH1], [HH2] for the definition and detailed study of test elements, 
and sec also Definition 1.6 for a generalization to the notion of a-test elements. 
Proposition 1.5. Let R be an equidimensional reduced excellent ring of charac- 
tertic p > 0 and let a be an ideal. Then for any parameters x,..., x in R, 
(«,. 
Proof. Actually, we can prove even more. Namely, let I, J Ç R be ideals generated 
by monomials in parameters x,..., x, and let I¢ ç R be file "expected" answer 
for I  ,1. that is, the monomial ideal that would be equal to I " J if Xl,..., 
formed a regular sequence. (Note that I = K = (Xl,...,x-a) and J = (x) in 
our case.) Then we will show that 
i. . J ç tç*. 
Let z  I*«  J. Then there is a c  R  such that czq q ç I[ql  jM for q = p >> 0. 
Since t.he "expected'" answer for IM  J[q] is K [q], We have that IM  J[q]  (N[q])* 
by colon-capturing of the usual tight closure [HH1]. So, for a test element d  R , 
one has that (cd)zq q ç K [q] for q = p >> 0, which means that z  K *, as 
required. 

Definition 1.6. Let R be a Noetherian ring of characteristic p > 0 and let  be 
1,11 ideal of R. \Ve say that an elelnent c  R ° is an c-test elemet if czqc q C_ I[ql 
for all q = pe whenever z  I *. 
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In the case where a = R is the unit ideal, an a-test element is nothing but a test 
element in the usual sense [HH1]. In [ItH2] it is proved that a test elelnent exists 
in lmarly every ring of interest, for example, in excellent reduced local rings [HH2, 
Theoreln 6.1]. We Call show that ail a-test elelnent also does. 
Theorem 1.7. Let R be a reduced Noetleria, ring of characteristic p > O. let 
c E R °, ad assume that one of the following conditions holds: 
(1) R is F-Jïite and the localized ring Re is strongly F-regular; 
(2) R is an ezcellent local ring and R« is Gorenstein ad F-regular. 
Then some power c n of c is a a-test clentent for all deals a C_ R. 
Here we provc thc abovc theorem mder assumption (1) only, according to the 
mcthod of [HH0]. The case of assumption (2) is reduced to the F-finite case by 
the machinery of "F-construction" used in [HH2]. We do hot include the argument 
involving this reduction process, bccausc it is somewhat long but essentially the 
samc as that for the usual tight closurc [tlH2, Section 6]. 
To prove thc theorcm in the F-finitc case we need the following lemma from [HH0, 
Remark 3.2], in which it is impliçit that thc eponent t of c may be independent 
of the choice of d, 
Lenlma 1.8. Let R be an F-finte reduced Noetherian ring of characteristc p > O. 
If the localization R« of R af an element c  R ° is strongly F-regular. then there 
eJ:ists an integer n _ O, dependStg only on R and c. satisfying the following property: 
For any d  R °, there ezist a power q of p and an R-linear ntap dp: R /' -- R 
sending d /' to c n. 
Proof of Theorem 1.7 in case (1), We will show that c n in Lemma 1.8 is an a-test 
clement for every a C_ R. Given any idem I. anv z  I *« and any power q of p, it 
is enough to show that cnzqa q C_/[q]. Since z  I*«, there exists d  R ° such that 
dzQa Q Ç I [QI for every Q. Then by Lemma 1.8, there exist q' and " R /' -- R 
sending d 1/' to c'. Since dzqq'(aq) [q'l ç dzqq'aqq' Ç I['], one bas 
dl/q'zqaqR /' C IMR 1/'. 
and applying ¢ gives cnzqa q C IM, as required. [] 
Proposition-Definition 1.9. Let R be a Noetherian ring of characteristic p > 0 
and let a be an ideal of R, Let E = (m ER(RIra), the direct sure, taken over all 
mazimal ideals m of R, of the injective envelopes of the residue fields R/m Then 
the following deals are equal to each other, and we denote thern by v(a): 
i) N AnnR(0ï), where M runs through all finitely generated R-modules: 
II 
ii) N AnnR(0ï), where Iii runs through all fln.itely generated R-submodules of 
MC_E 
E. 
If R is locally approzimately Gorestein ( e.g., if R is ezcellet ad reduced [Hol]), 
then the following ideal is also equal to -(a)  
iii) Ç (I  I*), where I runs through all ideals of R. 
_R 
Pro@ The proof is the saine as that for the usual tight closure. See [HH1, Proposi- 
tion 8.23] for the equality of i) and ii), and [HH1, Proposition 8.25] for the equality 
of ii) and iii). [] 
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Remark 1.10. In the case where a =/ is the unit ideal, r(a) = r(/ï') is called the 
test ideM. In this case, -r(R) al R ° is equtl to the set. of test elements of R, and this 
justifies the naine "test ideal". But tlle naine "a-test ideal" for r(a) is somewhat 
nfisleading if a :/- R. because r(a) C R ° is hot equal to the set of a-test elelnents in 
general. 
The following basic properties of the ideal "r(a) follow from Proposition 1.3. See 
Theorem 2.1 for a generalization of the latter half of (1). 
Proposition 1.11. Let R bc a Noetherian ring of characteristic p > 0 and let n 
and b denote ideals of R. 
(1) r(n)b C_ v-(ab). Moreover. if b is « principal ideal of a complete local ring, 
then T(a)b = r(ab). 
(2) If b ç a. then r() Ç r(a). /1loreover, if arqR °  1? and if  is a reduction 
of a, then r() = r(a). 
(3) /f R admits a test elerret and if 
(4) If R is weal,'Ig F-regnlar, then a ç r(a). /1loreov«r, if a is an ideal of pure 
height o'ne, the a = r(a). 
Proof. The first half of (1) is immediate ff'oto Proposition 1.3 (3). To prove the 
second hale let (R,m} be a complete local ring and let  be principal. Then bv 
the Matlis duality, AnnE(r(a)) is equal to the ration of 0î taken over ail finitely 
generated submodules ai of E = En(Rira). So, if z  Annt¢(r(a)b), then there 
exists a finitely generated submodule AI C E such that z  (0Î  b)t¢. Replacing 
M by M + R: C E, one has z  (0Î  b)^l = 0ï bv Proposition 1.3 (3). Hence 
r(nb) = N AnnR(0î  b)M = AIlIIR(AIIIIE(T(fl)b)) = T(a)b. 
(2) follows froln Proposition 1.3 (4), and (3) and the first halfof (4) from r(R)n _C 
r(n). As for the second hall of (4), it sufiïces to show the following claire, since 
weakly F-regular rings are normal. 
Claire 1.11.1. lf R is normal and n is an ideal of pwre height ore. then "r(n) C_ n. 
To prove the claire, considering a primary decomposition of n, we may assume, 
without loss of generality, that n is a primary idem such that p = v/ is a height one 
prime ideM. Then. since Rp is a discrete valuation ring, we can choose b  n such 
that n/ï'p = b/ï'p. Then bR  n C n *a. Indeed, if z  bR  n, then zVn  C bV/ï ' _C nM 
for all q = p, so that z  n *a. It now follows from bR  n OE p that n *a OE p. Since n 
is p-primary, we have T(n) C n " n *a = n. as claimed. [] 
Proposition 1.12 (cf. [BI, [HH1, Proposition 4.12]). Let I:¢ Ç S be a pure ring 
extension of Noetheria rings of characteristic p > 0 such that R ° C_ S °. Then for 
any ideal n of R. one bas r(nS)  R C_ r(n}. 
Proof. For a finitely generated R-module ]lI the natural map M = ]I @ R -- 
M @ S is injective by the purity of R C_ S. Since R ° _C S °, we see easily that 
Oî C .8 via the inclusion map 1I - ]I @n S. Hence, if c  r(nS) ç R, then c 
-- 'M®S 
kills 0î for ail finitely generated R-modules /11. so that c  r(n). [] 
By definition, the idem r(a) is the annihilator of a-tight closure relations for ail 
ideals or filfitely generated modules. It will be verv useflfl if r(a) is deternfined by 
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a-tight closure relations for a single idem or a single module. Let us take a look at 
SOlne cases where this is truc. 

Pro@ 
skel <'h 

Theoreln 1.13. Let (R. m) be a d-dine-n»ioTml excellent normal local ring of char- 
acteristic p > O, a an ideal of R. and let J C R be a divisorial ideal such that the 
divisor clas.s cl(J) E CI(R) bas a finite order. Then 
MCHg(J) 
n,here I rurs through oll fiitel9 geerated R-submodules of H(J). In paicular, 
if R is Q-Goresteim then 
(a) 
= Aml(0« ), 
where E = En(Rira)  
Again thc proof is the saine as that for lhe usual tight closure,  but we 
a proof according to [Sm2, l,elmna 3AI, which is bmsed on lhe idea of [AM, 

Let r be lhe order of ci(J) E CI(R) and let ,1 ) = Xl]ï'. We mav assume, without 
loss of gelmrality, that .r G m. Then there exist x2  R and 0 ¢ a G J such that 
weJ ç OR, ald xt,xe eXleldS [o a 83"Slelll of pa«'ameters x,x2,..., Xd for R. 
The point of the proof is *hat F«(H(,I))  H(J (p)) is computed by 
( ) in/(,", 
whcre the direct limit map R/(x[Jq),x,...,x) 
is the multiplication bv xx2''.Wd. Then an element { G H(J) is represented by 
z mod (xïJ, xee .... ,x) G R/(xJ, xee,...,x) for some z 
[ ,,od (xJ,x ..... .a)] i ,,pp« to  = [« od (xï*J», x*,... «*)1 
by the e-rimes iterated Frobenius nmp F «" H(J) H(J{)). 
Now sv that { G 0(j). Then there exists c G R  such that cqa = [czqa mod 
qs pe 
(xïJ(q),x 2 ,...,x*)] = 0 for all q = » 0 and  G a . Sinçe a q is a finitely 
generated ideal for each q = pe, tllere exists 
(xï+ j(v -+« -*+, *++ 
"2 ,...,x d ) ç (xï ,x 2 ,...,ad /. Then one h 
czqa ç (xï +[q],x 2, .... x) * by colon-capturing. Repla«ing c by ce' with c' a 
test element and nmltiplying by x, we see lhat cxzqa q C (xïJ(q 
_ ,, ,... 
This gives 
_,+1) (,+1) _,+ _,+1 -,+I)M 
_,+1. Hence { is 
for an  = v >> 0, -hen« (Xl-..«) e (*+1.4+1 ..... .  . 
in the a-tight closure of zero in the cyclic (hence finitely generated) submodule of 
H(J) generated by the image of R/(xï+lj, x +1 .... ,x+l). 
Discussion 1.14. In Sections 2 and 5, we will consider when the equality r(a) = R 
holds. When R is a Gorenstein local ring, one can check this condition looking only 
at the a-tight closure of a single parameter ideal,  we will see below: cf. [FW]. 
Let (R. m) be a d-dimensional Cohen-Macaulv local ring, a any ideal of R. 
and let J be the ideal generated by a system of parameters Xl,...,Xd. Then 

1The proof in [Ha2, Appendix] has a minor gap at the bottom of p. 1904, although the result 
[Ha2, Theorem 1.8] itself and the arguments in the cited references [hic], [Wi] are valid. 
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H(R) - limR/(xî,...,xâ), and R/J aud H(R)have the saine socle in conlnlon 
via the im.tural inchlsion map R/J  H(R). Then 0(R ) = 0 if and only 
if 0j = 0 or, equivalently, if J*" = J. In particular, the condition that ,J is 
«-tightly closed does hot depeud on the choice of a parameter ideal J. 
Now assume further that (R, m) is Gorenstein. Then E = E(R/m)  H(R), 
*a X t ) . 
and one secs easilv that ç(a) = AnuR(0 d ) = tN(Xî, (Xî 
. H(n) "", a ,"" 
Therefore r(a) = R if and only if J*" = J for soIne (or equivalently, every) idem J 
generated by a system of pa.ramcters. 
As we bave seen so far, the «-tight closurc of the zero submodule in the injective 
envelope En(Rira) or the top local cohomology H(R) of a local ring (R. m) plvs 
a particularly important role.  close this section by the following proposition, 
which generalizes Smith's characterization of the usual tight closure of the zero 
submodule in H(R). 
Poposition 1.15 (cf. [Sml]). Let (R, m) be a d-dimensional excellent normal local 
ring of characteristic p > 0 and let  Ç R bc an ideal such tbat a  R   O. Then 
O(R ) is the unique matinal proper s«bmodu.le N with respect fo tb.e property 
aqF(N) ç N for all q = p, 
u,hee F " H2() H2() is te e-tiges iterted Frobeni,s 
Proof. Let c  R  be au element such that R is regular. Then c is also regular by 
the excellence of R. Hence solne power c  of c is an c-test element and an «-test 
elelnent., by Theorem 1.7. It is easy to sec that c n also works as a test element for 
both the «-tight closure and the «-tight closure of the zero sublnodule in H (R) = 
H() (see the proof of Theorem 1.13). Then it follows that 0n(n 
so we may sulne, without loss of generality, that R is a complete local ring. 
It is easy to sec that nqe/n*a _ 
.  -- tH(R)) C OH(R ) for ail q = Also, 0H(R ) 
proper submodule of H(R), bccause it is annihilated by r(a) by Theorem 1.13 and 
ç(a)R    bv Proposition 1 ll (3). To prove the maximality of 0*ç suppose 
 " H(R)' 
that N C H(R) is a proper submodule such that «qF(N) ç N for all q = p. 
Then tac Matlis dual of the exact sequence 0  N  H(R)  H(R)/N  0 is 
o  [H2(n)/n]   ,  v  0, 
where [H(R)/N] v is a nonzero submodule of n- So both [H(R)/N] v and 
are torsion-free R-modules of rank 1. Therefore N v is a finitelv generated torsion 
module, so that there exists c  R  such that eN v = 0. This implies that eN = 
eN vv = 0, so that «aqF(N) = 0 for all q = p. Hence N 
2. -TIGHT CLOSURE AND ITS APPLICATIONS 
In this section we give some fundamental applications of a-tight closure. 
Modified Briançon-Skoda theorem via «-tight closure. One of the important 
applications of tight closure theory [HH1] is a prime characteristic proof of the 
Briançon Skoda theoreln [BS], which was originally proved by an analytic method. 
Later, Lipman [Li] improved this in terres of adjoint ideals. The following is a prime 
characteristic analog of Lipman's "modified Briançon-Skoda" [Li, Theorem 1.4.1]; 
sec also Remark 3.2 (1). 
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Theorem 2.1. Let R b« a Noethcria'n ring of «haracteristic p > O. If a C_ Ris an 
M«al 9«n«rat«d b!l r clcm«nts, the71 
for ail t k O. If we assu'me furlher thal R zs a complete local ring. the 
for all n. Z 0 attd all ideals b ç R. 
Proof. Let b ç R l»c any id«al, AI any finitely gcncrated R-module, and suppose 
that z  (()ï  a"). Then there exists c  R  such that czq(an)[q]b q = 0 iii 
F«(,II) fin all q = t » >> O. Sin('c a (+-)q ç (a") lui bv the assumption, this 
.O nr-I  
implies that czq(an+r-lb) q = () iii F«(5I) for all q = p » O, so that z  0 M 
.an+r--I 
Thus (0ï  a")A C_ "An*«+- ami, in particular, AmM(a") C_ 0 M . Taking 
Ihe inIersc('ti(m of the mmihilaIor ideals over ail finiIelv generate(l R-submodules 
Nw u,. th«t (, ) i , c(,,m,(«(  ()«« ri.. n,e h..(()) = 
I)y thc la(lis duality, and it follows as in the latter half of Proposition 1.11 (1) 
_ _ n*«"+-o where the lHliOlS are 
that AnnE(r(b)a") C UMÇE(0ï " an)M C UAIÇE AI 
taken over ail finitoly generated submo«hfies I of E. Tlms we conchlde that 
(an+r-Ib) ç AnnR(O;î ".'t)M = AnnR(Al,nE((b}an)) = ç(b}a n. 

lcmark 2.2. The tight closure version of the Briançon-Skoda theorem [HH1. The- 
orem 5.4] says that if a is generated bv r elements, then a ri+r-1 _C (an) * for all 
, > 0, wherc b denotes the integral closure of an idem b. This implies that 
r(R)a'+ "- C_ a", and in the case where the test idem r(R) is a strong test ideal 
(this is the case if R is a reduced complete local ring [Vr]), r(R)a +-1 C r(R)a '. 
Theorem 2.1 may 1)e considered a slight improvemcnt of these a.ssertions, because 
r(R)a +- C r(a n+-l) = r(a '+'-1) 1)v basic properties of a-tight closure; sec 
also Discussion 5.2. 
Recelflly, using arguments silnilar to the above, the first author and S. Ta'l'kagi 
l)roved a sharpened version of Theorem 2.1 [HT]; cf. [Lil, [LI: if (R. m) is a complete 
local ring of characteristic p > 0 and if a is an ideal with a reduction generated by 
r elements, then r(a '+'-) = r(ar-1)a" for ail n > 0. 
Corollary 2.3. Let R be a reduced excellent rtng of characteristic p > 0 and let a 
bc a ideal such that a ç R °  O. Tte for aty R-modules N C M and any z  M. 
the following conditioTr are equivalett. 
 a _ N[ï] p. 
(1) z  N^. i.e.. there exists c  R ° such that czqa q C for all q = 
(2) There exists c  R ° such tbat czqr(a u) Ç Nï ] for all q = p. 
(3) There exists c  R ° such tbat czqa- C Nï ] for all q = p. 
Proof. To prove (1) => (2), choose d  a-çR ° and apply dr(a q) C r(aq +'-) C_ au. 
As for (2) => (3). choose a test element d  r(R)çR ° and note that d C_ r(aq). [] 
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Corollary 2.4. Let ( R, m) be a d-dimensional Noetherian local ring of characteris- 
tic p > 0 with infinite residue field. Then for any ideal a Ç R and for any n > O. one 
bas 7(a +d-) ç a . If, in addition, 
Pro@ First assume that a is an m-primary ideal. Then a has a minimal reduction 
q ç a generated by d elements, so that 
Next let a be an arbitrary ideal. Since our assertion holds truc for every m-prilnary 
ideal, it follows that 
tEN tN 
The second assertion is proved in a similar way. 
Tight integral closure vs. a-tight closure. First. we recall the notion of tight 
integral closure, which was introduced by ttochster. 
Definition 2.5 ([Ho2]). Let  be a Noethcrian ring, raid let {I, ..., I } be a set 
of ideals in R. An element  H is in the ti9ht inte9ml closure {I ..... I,}  if 
there exists c  R  such that cx q  i= Iï for ail suNciently large q = pe. 
In [HWY1], the present authors have studied the F-rationality of Rees algebras 
R(I) = R[It] for m-prinmry ideals l, jointly with K.-i. Watalmbe. One of the main 
rults in [HWY1] is the following theorem, which gives a criterion for F-rationalitv 
of Rees algebras in terres of tight integral closure. 
Theorem 2.6 (cf. [HWY1, Theoreln 2.2]). Let (,m) be an ecellent Cohen- 
Macaula normal local ring of chamcteristic p > 0 with ifinite residue fleld. Let 
I be an m-primar ideal of R and J its minimal reduction. Fiz an sstem of 
pammeters x .... ,xe for R generating J and put j[tl = («, ...,«é) for l  1. 
Then the Rees algebra R(I) = R[It] is F-rational if and onl if R(I) is Cohen- 
Macaula and the following equalities hold: 
{I dl-r, X,..., X} = I dl-r + J[/] /OT all l, T OE ] with 1  1"  dl - 1. 
We will show that all tight integral closures appearing in the above theorem can 
be represented as the form of some "-tight closure?  Namely, we bave the following 
theorem. 
Theorem 2.7. Let (R, m) be an excellent equidimensional reduced local ring of 
chamcteristic p  0 with d = dimR  1. Also. let x .... ,Xd be a system of 
parameters of R, and put J = (x, ..., Xd)R. Tben we bave 
(Jd-, xR ... ,xR} 
for ail integers l, r  1. 
Before proving the above theorem, we give some corollaries. çY now recall that 
 +---+  ç {h, ..., I}; 
s [Hol, Proposition 1.4]. 
Corollary 2.8. Using the saine notation as in Theorem 2.7, if J ç I ç J, then 
(j[l]).J   jat- + j[l] for all l, r  1. In particular, we bave 
(1) J+ I a-  J*. 
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(2) If .l *r = .1, then I a-' C ,1. 
As an application of Theorem 2.7, we can r('write Theorem 2.6 as follows. 
Corollary 2.9. Using the saine notation as in Theorem 2.6. the Rees algebra 
i.s F-rational if and only if R(I) is Çohen-Macaulay and the following equalities 
hold: 
(,l[t])*t = I'u- + .lit] for ail l, r  1 with 1 < r < dl - 1. 
Corollary 2.10. Let  b« an. c,r«cllent F-rotional local ring with dira H = 2. Then 
for any 1,arameter idcal .I of . u,c have ,]*J = . 
Pro«f. Put I = J. Thon .I is a minimal re(hwtion of I. Since R(I) is F-rational 
1)y [HWYI, Th('or('m 3.1], it folk)ws from Thcorcms 2.6 and 2.7 that 
[ + .l  I, as require(1. 

lu th(' folh)wing, we prove Theor('m 2.ï, and so we assume that (R, m. k) is an 
excellcat cquidimcnsional (hot ne('(,ssarily ('(hwcd) h)cal ring of characteristic p > 0 
with d = dira R  l. Also. let .r ..... xd bc a systcm of paramct('rs of R and put 
.l = (Xl ...... ra)R. Further, we s(,t .l[q := (.ri .... r) for all l  l. 
Lemma 2.11. Snppose that x ..... xa form a regular sequençe. Then for ail in- 
tcgers l. r  ] we bave 
(.,. .... )-(, .... ,,)" = (., ..... .)'-'-'+' + (. .... 
thot s. .1 [t]  .1"= jdt--d+ + 
Pro@ The right-hand side is contained in the left-hand side because jat--d+j,, = 
jd(t-D+ ç j[t]. We nmst show thc opposite inclusion. To do that. let w = 
xï.--x , whcre 0  a,  l - l for all i. First suppose that Zî= ai = dl - r- d. 
If we put bi = ! - 1 - ai for all i, thel, b i  O. Zî=l bt = ç and u'-xî'---,râ « = 
/--1 . . . 
.r  .l[q. Next suppose that ,. ai 2 dl - r - d + 1. Then for ail 
d c ca az+cz aa+ca j[l], 
integersci OE0withi=c=r, wehave w.x 1 ---z a =z 1 ""xa 
d 
hecause i=(ai + ci)  d(l - 1) + 1. Since ,l[q  J" is generated bv monomials in 
.r .... , xd, the a.ssertion follows frOlll the above argullmnt. 

Using the colol-capturing property of tight ch)sm'e, we obtain the following. 

Corollary 2.12. In. the above notation, for all l, r >_ 1, we bave 
,IN  jr ç (jdl-r-d÷l + j[/]).. 
Pro@ First suppose that R is complete and reduced. If we put S = k[[x .... , xa]], 
then S is a complete regular local domain and R is a finitelv generated torsion-free 
S-module. Also, if we put Jo = (x, ..... za)S and J'] = (xî ..... x)S, then 
J = J0R and J[Z] = JZ]R. Using the colon-captnring property of tight closure and 
the previous lelnma, we get 
j[l] . jr ç ((jl, . j))" = ((4/-r-d+l + j;/]))* = (jdl--r--d+l + j[l])" . 
Next we consider the general ce. Fix I, r 2 1 and put K = jdl-r-d+l + j[q. 
Applying the above gument to d = R, we have 
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and hence .l[t]/..1" C (Iç)*. By [BH, Proposition 10.3.18], we get 
«tl..r = («Il. J")n R = (,,)* nR= ,ç*n  = I,*, 
as required. 

We are llOW ready fo prove Theoreln 2.7. 

Proof of Theorem 2.7. Note that R admits a test element c' E R °, because Ris an 
excellent reduced local ring ([HH2, Theorem 6.1]). 
Let z E (J[Zl) *. Bv detinition, there exists c" E R ° such that czqJ rq Ç j[lq] 
for all q = p, e » 0. Corollary 2.12 implies thal 
(' Z q  j[Iq] . jrq ç (j(dl-r)q-d+l + j[Iq])., 

and hence 

(" c"z q  .1 (dl-r)q-d+ l -k .1 [d 
for ail q = p. c » 0. Take tlly elen,ent c m  jd-1  Ro and put c = cccm  H o. 
Then oz q e j(dl-r)q+.][Iq] for ail q = p', ç >> 0. Thus z e {j,tl-r, w{ ..... £éH} . 
Next we prove the opposite in«htsion. Let w e {,la-, a'R .... ,,réR}. Bv 
definition, there exisls «"  R ¢ snch that c"w q  j(dl-r)q+,l[lq] for all q = p, e » 0. 
Thus ('ttwqJ "q C j,llq + j[Iql. 
On t he ol her hand, bv virl ue of t he t ight clostre Briançon -Skoda t heoreln [H H 1. 
Theorem 5.4], we have 
.ldlq ç .jalq Ç (j[lq]).. 
Taking a test element «'  H , we have c' (,j[lq])* Ç j[lq] for all q = p, e » 0. 
In particular, we have cctwqJ rq  j[Iq] for all sucientlv large q = p, and thus 
w  (J[/])*J. as required.  

Remark 2.13. Although we tan easily see that ,l*Red C_ (JRred) *IRra alwavs 
holds, we do llOt klloW whether or llOt the opposite inclusion holds. If it were true, 
we could remove the aSSUlnption that "R is reduced" iii Theorem 2.7. 

A characterization of regular local rings. Let (R, m, k) be an excellent equidi- 
nlensional reduced local ring of characteristic p > 0 with d = dinl/¢ _> 1. Then/ï' is 
F-rational if and only if J* = J for SOlne (every) paranleter ideal J of R: sec [HH2] 
and [FW]. 
Let J be a lnininlal reduction of m. Since ,1" = j.n. we have the following 
increasing sequence of ideals iii/ï': 
J c J* c ,1 *' c J*' c ... c J *md-1 C J*' = R, 

where the equality on the right follows from the tight closure Briançon-Skoda the- 
oreln ([HH1, Theorem 5.4]). So it is natural to ask the following question, 

Question 2.14. Let R be ai1 excellent equidilnensional local ring of characteristic 
p > 0. and let J be a minimal redtmt.ion of m (iii general, a parameter ideal of R). 
}Vhen does the equality j.,-i = J hold? 

In the following, we give a characterization of regular local rings in terres of 
-tight closure, which gives an answer to the above question. See also Section 5 
about re|ated problems. 
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Theorem 2.15. Let (R, m, k) be an excellent equidimensional reduced local rang 
of characteristic 1) > 0 with d = dimR _> 1, and assume that k is inJïnite. Then 
the following conditions are equivalert: 
(1) R is regular; 
(2) T(m d-l) -- R. i.e., I *md-' -= I holds for every deal I of R: 
(3) J *md-' = J holds for some parameter ideal J of R. 
In ordcr to prove Theorem 
hot need to assulne that R is 
Lemma 2.16. Let (R,m) be 
I *m'- = I for every ideol I of 
Pv@ Suppose that I *m'- 7 
definition, there exists c ( R ° 
Sin('e the Froi)enius mal) F" R 
«m(d-1)q 
and hence 
C tE m [q]  m (d-1)q = m q-d+l 
for ail q = p, e » 0, by Lemma 2.11. This is a contradiction. [] 
Proof of Theorem 2.15. Note that R is approximately Gorenstein by out assump- 
tion. (1) == (2) follows from Lemma 2.16. Also, (2) == (3) is trivial. 
Let us prove (3) := (1). Take a parameter idem J such that J*ma- = J. 
Since J C_ J* Ç J *ma-I -= J* 
, we bave J . Hence R is F-rational, and thus is 
Cohen-Macmflay. Then for a mininml reduction q of m, we have q.a-, = q (see 
1.14). Thus we may assume that J is a minimal reduction of m. Then bv virtue of 
Corollary 2.8, we have m c .1. This implies that R is regtllar, as required. [] 

2.15, we need the following lemma. Note that we do 
excellent in the proof of this lemma. 
a regular local ring with d = dimR _> 1. Then 
R. 
I for some ideal I of R. Let z  I*- \I. By 
such that czqm (d-1)q ç I [ql for ail q = p, e » O. 
 R is fiat bv Kunz' theorem ([Ku]), we have 
C_ I [ql  z q = (I" z) [ql Ç m [q], 

3. INTERPRETAT1ON OF MULTIPLIER IDEALS VIA a-TIGHT CLOSURE 
We posed Theorem 2.1 as a prime characteristic analogue of Lipman's "modified 
Brianç'on-Skoda theorem'" [Li]. The original form of "modified Briançon-Skoda'" is 
stated in terres of what is called "adjoint ideals" by Lipnmn. 1Recently. this notion is 
reformulated in the theorv of "multiplier ideals" from a different point of view and 
plays an importmt role in birational algebraic geometry; see [Ei], [La]. Actually, 
one can define multiplier ideals with "'rational coetficients"; cf. Section 6. 
Definition 3.1. Let " be a normal Q-Gorenstein variety over a field of charac- 
teristic zero and let a C O:« be a nonzero ideal sheaf. Let f" X --, }" be a log 
resolution of the ideal a, that is, a resolution of singularities of Y such that the 
ideal sheaf aOx is invertible, say, aOx : Ox(-Z) for an effective divisor Z on X, 
and that the union Exc(f) U Supp(Z) of the f-exceptional locus and the support of 
Z is a simple normal crossing divisor. Given a rational nuinber t >_ 0, the multiplier 
ideal (.7(t. a) = (.7(a ) associated to t and a is defined to be the ideal sheaf 
ff(t. a) = f.Ox([Kx/r - tZ]) 
in Or, where the Q-divisor Kx/y = Kx -f*Içy is the discrepancy of f. For t = 1 
we just denote ,27(a) := ,27(1  a). This definition is independent of the choice of a 
log resolution f: X --, }" of a. 
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Renmrk 3.2. (1) If b = t ' for a nonnegative integer n, thon ff(t. b) = ff(t. «), and 
this justifies the notation ff(et) with "fbrmal exponent" t. Henceforth we prefr 
the exponential notation ff(t) rather thaa ff(t  ): cf. Section 6. 
(2) Multiplier ideal ff() bave proiwrties imilar to those of the idcals r(); sec 
Proposition 1.11. Namely, if b ç , then ff(b) ç ff(), and if b is a reduction 
of , then the equalit.y ff(b) = ff() holds; ff()b  ff(b) for any . b, and if b 
is locally principal, thon the equality ff(«)b = ff(ch) holds; ff(c) # 0 as long as 
« # 0; and if " bas only log terminal singularit.ies, thon « ç ff(c). Also, using 
vanishing theorems in characteristic zcro, one can prove a "'modified Briançon- 
Skoda theorem'" ([Li, Theorem 1.4.1], [Lai): If a is generated by r elements, then 
ff(+-l) ç  for every   0: cf. Theorem 2.1. La.ter in Section 4, we study 
more about similaritv of the ideals r() and ff(). 
3.3. Reduction to prilne characteristic. It is proved in [Ha2] and [Sm2] that 
the multiplier ideal ff(R) of the unit ideal in a normal Q-Gorentein ring R essen- 
tially of finite type over a field of characteristic zero coinçides, after reduction to 
charactcristic p » 0, with the test, ih,al r(R). XX geueralize this result in Theorem 
3.4 below. To state the result, we ha.ve to ],egin with a ring R and an ideal « in 
characteristic zero, ami reduce them to characteristi« p » 0 together with a log 
rolution f: X  Spec R of . 
Let R be an algebra essentially of finite type over a field k of characteristic 
zero, and let  ç R be an ideal. One can choose a finitely generated Z-algebra 4 
contained in k and a subalgebra RA of R essentially of finite type over A such that 
the natural map RA @A k  R is an isomorphisn and « 4 = «  R,4 generates the 
ideal a of R. For a maximal ideal t of A. we consider the base change to its residue 
field  = (p) over A to get a prime «haracteristic ring R = HA @A U and an ideal 
 = .4R. The data consisting of  = n(#), R, a is considered to be a "prime 
characteristic model" of the original data in characteristic zero, and we refer to 
such (, R, a) for maximal ideals p in a suitable dense open subset of Spec A as 
"reduction to cha.racteristic p » 0"" of (k, R, a). Furthermore, given a morphism of 
schemes essentially of finite type over k (and even more, a conmmtative diagram 
consisting of a finite collection thereof), e.g., a log resolution f" X  Spec R of a. 
we can reduce this entire setup to characteristic p » 0. (Sec [Hal], [Ha2], [HH3], 
[Sml], [Sm2] for more details.)  use the phrase "'in characteristic p » 0"" when 
we speak of such a set.up reduced from characteristic zero to characteristic p » 0. 
The lnain result of this section is the folloxving theorem, which enSUl'm the cor- 
rmpondence of the ideal r(a) and the nmltiplier ideal ff(a). Sec Theorem 6.7 for a 
generalization of this theorem t.o the case of "rational coefficients". 
Theorem 3.4. Let R be a normal Q-Gorenstein local tin9 essentiall9 of finite 
tgpe over a fleld, and let a be a nonzero ideal. Assume that a ç R is reduced 
from charactestic zero to characteristic p » O. together with a lo 9 resoltion of 
singularities f: X + Y = Spec R such that aOx = Ox(-Z) is invertible. Then 
(.) = u(x.ox([x/-] - z)). 
3.5. The remainder of this section is devoted to proving Theorem 3.4. Our strategy 
is to reduce to the case where the ring R is quasi-Gorenstein by passing to a 
canonical covering; sec [T] for a direct proof which does hot use a canonical covering. 
Let (R, m) be a normal Q-Gorenstein local ring with a canonical module , and 
let r be the least positive integer such that the rth symbolic power w ) of w is 
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isonlorl)hic to /. (iven a fixed isolnorphisnl w(R r)  R, one has a natnral ring 
-1 )  s) 
sçructure of N = =0 w . . This is a quasi-Gorenstein local ring (i.e., ws = 
with the ma.xilnal ideal m Ç, ,=x w ), and we call S a canonical coverig of R. 
The following two lelllnla8 make it possible to reduce the proof of the theorenl 
to the quasi-Gorenstein case. 
Lemma 3.6. Let (R. m) be a normal local tin9 of choracteristic p > O, S a canonical 
coverzn 9 of R as above, and assune that r is hOt divisible by p. Then for any ideal 
aCR. 
r(aS)  R = 
Proof. hlentical t,, thc case where a = R; sec [lta2, Section 2], and also [Sm2].  
The following lclmna is also proved entirely  in the saine way  the ce where 
a = R [Sm2, Proposition 3.2], but we inçlude the proof for the sake of completeness. 
Lennna 3.7 (cf. [Slnl]). Let (R, m) boa normal local tin9 essentiallg of finite type 
over a field of chara«'teristic zero. and lct S be a canoical coverin 9 of R as above. 
Then, for ag id«al a Ç R and any rational number t  O. 
J((a»')')   = J(a'). 
Pro@ Let n: Spec.ç  Spec R be the canonical covcring, and let f" X  Spec R 
and 9: "  Spec,ç be log resohltions of a and aS. respectively, which lnake the 

following diagram commute: 

"   SpecS 
X f  Spec R 

Let E be the reduced divisor on X supported oi1 Exc(f)USupp(Z) and let G be the 
reduced divisor with the saine support as ir*E. Then the log ranfification formula 
(sec, e.g., [Ka]) tells us that 
(3.7.1) Içws + G = ir*(Kx/l + E) + P 
for some effective divisor P on l" such that codim(#(P), X) _> 2. 
Now let u E ,7(a ) = H°(x, Ox([h-x/n-tZ])), i.e., divx(u)+[Iç\-/R-tZ] >_ O. 
Since Supp(Nx/n - tZ) C_ E, one has Kx/ - tZ + E >_ [KX/l - tZ], so that 
divx(u) + fçwn + E - tZ >_ O. 
and this is a strict inequality for the coefficient in each irreducible component of 
/ç. Pulling this back by ir and applying (3.7.1) give an inequality 
div:-(u) + Iç-/s + G - t#* Z >_ 0, 
which is a strict inequality for the coefficient in each irreducible component of 
G. Since G is reduced, it follows that div:-(u) + [KWs - tir*Z] >_ O. Hence 
. e 4°(x, or([I,r/s - tir*z])) = :7((as)). 
Conversely, let u  7((aS) t) C R and fix an3" prime divisor D on X. To prove 
tt G J(at), it is enough to show that a := vo(u) + coeffz)([Içx/n - tZ]) is non- 
negative. If D is not f-exceptional, this follows because ir is étale and finite at the 
generic point of D and divr(u) + [Kws - tïr*Z] >_ 0 bv u  7((aS)t). Now let 



GENERALIZATION OF TIGHT CLOSURE AND MULTIPLIER IDEALS 3159 

D be f-exceptional, F any prime divisor Oll I" donlinating D (this in particular 
inlplies that F OE Supp(P)), and let. e := coeffF(#*D). Then 
(3.7.2) VF(U) + coeffF(ir*(fIçxl --/Z])) -- ea. 
Oll the other halld, since divy(a) + [Içy/s - tir*Z] >_ O. it follows frolll (3.7.1) tllat 
divy(u) + [ir*(K¥/n + E) - G + P - tfr*Z] >_ O. SillCe F OE Supp(P), this ilnp]ies 
that 

(3.7.3) VF(ll) + çoeffF([*(h'\'/R -- tZ)] ) > coeffv(-ir*E + G) = -e + 1. 
It follows ri'oin (3.7.2) and (3.7.3) that ca _> -e + 1, so that a > 0, as required. 

Remark. The proof of Lenmla 3.7 works hot only for canonical coverings but also 
under a weaker assmnption that R '--* S is finite and étaie in codimension 1. It is de- 
sirable that Lemlna 3.6 also holds for filiite extensions that are étale in codilnelision 
1. and this issue has becn recently settlcd by Takagi ([HT], [T]). 

Proposition 3.8. Let ( R, m) be a d-dimensional normal local ring of characteristic 
p > O. and let a be a nonzero ideal. Lct f: X  Spe«R he a proper birational 
morphism front a normal scheme X such that aOx = Ox(-Z) is an invertible 
sheaf, and denote the closed fiber off b,q E = f-(m). Then one bas on indu.sion 
where 6" H(R)  HÉ(Ox(Z)) is an edge map H(R)  Hoe(Ox) 4 the spectral 
d 
sequence H(Rf.Ox)  HË(Ox) bllowcd bg the nat,,ral map H(Ox)  
n(o.-(z)). 

Pv@ First note that, for anv q = pe and any c  a q C H°(x. O\-(-qZ)). we bave 
the following conlnmtative diagram with exact rows: 

Then aqF«(Ker(5)) C_ Içer(5) for ail q = p, and the COllChlsioll follows frolll Propo- 
sition 1.15. [] 

By virtue of Lenlmas 3.6 and 3.7, it is sufl]cient to prove Theoreln 3.4 in the case 
where R is quasi-Gorenstein, i.e., wh  R. Iii this case, however, the assertion of 
Theorenl 3.4 coincides with the following. 
Theorem 3.9. Let (R, m) be a d-dimensional notre.al local ring essentially of flnite 
type over a field of characteristic p, and let a be a nonzero ideal. Assume that 
a C_ R is reduced from characteristic zero fo characteristic p » O. together with 
a resolution of singularities f: X - I- = SpecR such that aOx = Ox(-Z) zs 
inversible. Then 
0/( m = Ker (H(R)- H(Ox(Z))). 
where E zs the closed fiber of f and 6 is the edge map as zn 3.8. or dually. 
Allnn (O/(R)) -- H°(x, oy\-(-Z)) irt a R. 
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Proof. First, let us discuss the situation in characteristic zero hefore reduction to 
characteristic p » O. In characteristic zero, we choose a nonzero element c 6 a 
such that R« is regular, and a log resolution f- X -- Spec R of the ideal ca. Then 
aOx = Ox(-Z) for an effective divisor Z Oll X, and Supp(Z+div,¢(c)) is a simple 
normal crossing divisor. We choose an f-mnple f-exceptional Q-Cartier divisor D 
and a suflïciently small rational numbcr e > 0 such that [+ediv,(c)J = Z, where 
2=Z-D. 
Now we reduce the entire setup as above to characteristic p » O, and switch the 
notation to denote things after reduction modulo p. Let the ideal a be generated by 
r elements. Since Re is regular, some power c s of c (Ç a) is a usual test element and 
also an a-test element. Also, sance -2 is f-ample,/C = (,>o H°(X,<x(-[nHJ))is 
a finit.ely geuerated module over R = (,,>o H°( X" Ox-(-nH)). Say K is generated 
in degree _< no. 
Since we are v«orking in characteristic p » 0, the «-tiInes iterated Frobenius map 

F  HOe(O.x (2))  Hoe(Ox(p(H + edivx(c))) 

is injective for ail q = p bv Proposition 3.6 and Corollarv 3.8 of [ttal]; see also 
[Ha2, Discussion 4.6] and [MS]. This implies that the map 

c.'F  Hê(Ox(Z))= H(Ox(H))-* H(Ox(p«2)) 

is injective for ail sufiîcientlv large e Ç N such that p% _> rn := r + 2s + no - 1. For 
such q = p » 0, we consider the following comnmtative diagram with exact rows: 

0  Ker() + H(R)  HOe(Ox(Z)) 
0  Ker() Ht(R) __m_ Hî(Ox(qH)) 

Croc q ¢ Ker(e) : AI,nH(R ) tt°(X, wx(-[q2])) 

for ail q = p » 0 by the above conmmtative diagraln. Then for ail suflïciently 
large q = p (_> 'no), there exists an integer  with 0 _< n <_ no such that 

crç  t [0" H°(X, Ox((n-q)2))]n(m, 

since/C is generated in degree _< no as a graded T/-module. Hence it follows from 
H°(X, Ox((n-q)2)) C_ H°(X, Ox,((n-q)Z)) C aq-'o that crn(q¢l q-r° ¢ O. On 
the other hand, since c  a and c s  r(R), 

by the tight closure Briançon-Skoda theorem. Thus we have that cçqa q # O. But 
this implies that ç  0}(m, since c  is an a-test element (cf. Theorem 1.13). 
Consequently, we have 0}( m ç Ker(). The reverse inclusion follows froIn 
Proposition 3.8, and we are done. [] 
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4. PROPERTIES OF THE IDEAL T(fl) ANALOGOUS TO MULTIPLIER IDEALS 
In this section we prove some properties of the ideal T(n) analogous to those 
of the multiplier ideal (a), which are round in [DEL], [How], and in Lazarsfeld's 
lecture notes [Lai; see also similar results for "tight closnre for pairs" in [T]. The 
results in this section are proved in any fixed characteristic p > 0. However, in 
view of Theorem 3.4, we can also soEv that they provide characteristic p proofs of 
the properties of multiplier ideals in characteristic zero. 
We also want to remark that the results in this section can be generalized to 
"rational coecients" (see Section 6), just like those for multiplier ideals. 
Theorem 4.1 (Restriction theorem, cf. [La]). Let (R, m) be a normal Q-Gorenstein 
complete local ring of characteristic p > 0 and let x  m be a non-zero-divisor of 
R. Let S : R/xR and assume that S is normal. Tben for any ideal  of R, 
(«S)  (a)S. 
Proof. Let ER = ER(RIra) and Es = Es(S/mS) be the injective envelopes of 
residue fields of R and S, respectively. Then one h Es  (0  X)En C En. We 
first prove the following claire, viewing Es as a snbmodulc of En via this inclusion. 
Claire 4.1.1. OË  Es C 0 *es 
 ES " 
Prooy of Claire 4.1.1. Since  is normal, we can choose an a-test element c whose 
image  in Sis nonzero by Proposition 1.7. Then we have the following commutative 
diagram for each q = p (see the proof of [HW, Theorem 4.9]): 
The map a Es  E is the inclusion map mentioned above. Note also that 
 r rd, (q), « E  d-1 q . [Wa], and  
F(E) = m[w ) andFs(s) = Hm (w)) by  F(Es)  F(E) 
arises as a connecting homoinorphism of the long exact sequence 
Hd-1, (q)  d-1 d-l, (q)  d, (q) d, (q), 
 t   ())  ws)  )£ 0 
amociated to 0  wÆ )  w )  w(q)R /xw (q)R  0 ([HW]). It then follows that 
 tw )/xn w  is a proper S-subInodule of n tw s ). Hence 
we can show as in the proof of Proposition 1.15 that Ker() is annihilated by an 
element   S . 
Now let ç  OË  Es. Then çF(ç)a  = 0 in F(E) for ail q = p, since c is 
an a-test element. This implies that F(ç)a q ç Ker() by the above commutative 
 ,aS 
diagram. Therefore F(ç)(aS)  = 0 for ail q p with   S , whence ç  "Es , 
as claimed. 

We continue the proof of Theorem 4.1. Since R is complete, 0Ë = (0  
so that 
 a (T(I:I) ÷XR) = (O'T(a)S)Es. 
OEn(qEs-= (O''r(a)+XR)En--- O xR Es 
Hence by Claim 4.1.1, we conclude that T(aS) C_ Anns(0Ë N Es) = T(a)S. [] 
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Rentark 4.2. Theoreln 4.1 implies that. F-regularity "deforms'" in Q-Gorenstein 
rings [AKM]. Namely, if Ris Q-Gorenstein, .r  mis a non-zero-divisor and S = 
R/a'R is F-regular, thon Ris also F-regular. This result fails in the absence of 
Q-(;orensteimmss [Si]. On the other ha.nd, the F-regula.rity of R does lot implv the 
F-regularity of S = /.r, and this suggests that the containment r(aS) Ç r(a)S 
is far ff'oin an equality in general; sec also [HXV. Theorenl 4.9]. 
()ur next objective is to apply Theorem 4.1 to show the property of r(a) called 
"subadditivity" (Theorem 4.5), whi«h was established for multiplier ideals in regular 
rings bv [)emailly, Ein and La.zarstk'ld [DÈL]. Our strategy is to mimic the idea of 
"'restriction te) the diagonal" usod in [DEL]. XX first prove the following fact. 
local ri,,vs u,ith resi&e fiel«l k. a,,d let T = R  S = k[[a' .... ,z, V,-.-. V,]] be 
tteir complet« tcsor prodtct. 
(  s)  (  ) = (  )T i., T. 
Proof. Xb regard  and S as sul)rings of T via the natural ring holnomorphisms 
B  T and S  T. Then what is tt) lu, proved is that aTO bT = abT. To prove 
this we mav assume, without loss of generality, that t is a proper ideal of S. 
First we note that the composition   ç  T/bT of the natural maps is a fiat 
ring extension. Ind«ed.  =  k "   @k ,ç/b is fiat, since "  S/b is fiat by 
b  k = 0, and the conlpletion mai»   S/b  T/bT is also fiat. 
Now let F. bc a.n -free resolution of R/n. Then F. C'n T is a T-free resolution 
of T/T, sinco T is fiat over B. Since T/bT is also fiat over , one bas that 
Tory(T/T.T/T) = H,((F. On T) $T T/) = H,(F. n T/T) = 0 for i > 0. 
In pa.rticular, (aT  bF)/abT  Tor(T/aT. T/bT) = 0. Thus we conclude that 
aF  b T = ab T, as required. 
Proposition 4.4. Let " be a field of cha,'acte%,tic p, a,,d let  = k[[x ..... x]], 
4.3. Thez. for azz ideal 
Proof. Let us denote the injective envelopes of the residue fields of . S. T by 
En, Es, ET, respectively. Then we can describe them in terres of inverse poly- 
,o,,,i.l   = 
Er = (x ...xv ...V,)-[,7,...,,rT,Vï,....VZ]; so. in particular, Er = 
0*{a@b)T 
En  Es. Then it is easv to sec that 0oe  Es + En C. 0oeos Ç oe7 . Hence 
*a 
r((a )T) g AmT(0  Es)  Amer(En  0) 
= (.(a) $ s)  (n $ .()) = (.(a) $ ()) 
by Lemlna 4.3. 
Theorem 4.5 (Subadditivity, cf. [DEL]). Let (R.m) be a complete regular local 
ring of characteristic 
(a) ç (a).(). 
Pmof Let T = B  B, and let  T  B be the ring homomorphisni sending 
x @  Ç T to x  B. If we restrict the containment r(abT) Ç r{a)r(b)T in 
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Proposition 4.4 bv the diagonal map A. T -+ R. we immediatel. obtain 
by virtue of Theorem 4.1. 
Remark 4.6. If (R, m) is hot F-regular, then it is clear that the subadditivity breaks 
down for a =  = . ç do hot know whether or not the subadditivity always 
holds in F-legular rings in genelal. 
4.7. Toric case. In [How], Hovald gave a combinatolial description of the lmflti- 
plier idem J(a) of a lnOnomial ideal a in a polynomial ring over a field.  show 
that the ideal ç(a) has a similar description in a more general situation, nalnely, a 
is a toric ideal of a toric ring  over a fiehl k. Note that, in this case, the imllti- 
plier ideal (a) can be defined even if char k = p > 0, because there exist.s a log 
l'esolution of a in the toric category. 
Let fil = Z d, N = Holnz(/, ), and denote the duality pailing of 
with N = N@zN hy ( , }" .1 x N,  N. Let  C  be astrongly convex 
l'ati¢mal polyhedral cone, and dem,te «v = {,n, 
usual. Let R = k[ v  «lI] be the toric rillg over a fiold k defined by , that is, the 
sui»ring of a polyliomial ring k[ah,..., md] generated as a k-algebra by lnOnoinials 
 rm = a'î  " " " d'm with m = (ml,    , mE)  «v  M. Also, let D, .... D» be the 
toric divisors of Spec R corresponding to the primitive generators ni, .... ns  N 
of o, respectively. A toric ideal a ç R is an ideal of R genel-ated by mOnOlnials 
iii Zl,..., Xd. Let a Ç R be a toric ideal and let P = P(a) c Ie be the Newton 
polygon of a, that is, the convex hull of {m  Al] .  a} in M. We denote the 
relative interior of P in [ by Int(P). 
Now a.ssmne that  is Q-Goi'eislein. Then there exists w  Ie such that 
(w, ni} = 1 for i = 1 ..... s. IiMeed, since w r) is principally generated RI" some 
r  N and w ) corresponds to the divisor -r i=1 Dt, we tan write w ) = 
for some m0  M such that (m0, ni} = VD, () = r. Then set w = mo/r  &. 
Theorem 4.8. Let R = k[o v  3I] be a Q-Gorestein toric ring over a field of 
characteristic p > O. ad let w  Ma be as above. The. for 

Moreover. for m  M, the following conditions are 

and it is again a toric ideal. 
equivalent to each other: 
(1) x m  "r(a): 
(2) m + w  Int(P(a)); 
(3) 
Pro@ We prove that 7-(1:1) is generated bv lnonolnials x' satisfying the condition 
m+ w  ïnt(P(a)) in (2). It is essentially proved in [How] that ff(a) has the saine 
property. 
First. to simpli" out argument, we note that 1  R ° is an a-test element, because 
toric rings are strongly F-regular. Honte, an elelnent z  E of the injective envelope 
E = En(Rira) of the residue field R/m = k is in 0Ë  if and only if zqa q = 0 in 
F(E) = cH @ E for all q = p. 
Next we will compute the FrobelfiUS map F: E -- F(E) explicitly. To do this 
we note that FO(E)  ,,a, (q), 1) is k-dual to 
= mtcoR ) for q = pe by [}'al] , a.lld Hdm(02 
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c°( -q = ((m,n,)>l--qk . x TM. Therefore, 

(m,ni)<q--1 rnE(q-1)w--« v 

and thc Frot,enius map Fe: E -+ Fe(E) sends x TM E E to x q  Fe(E). 
It is now clcar that 0Ë a and hcnce "r(a) = AnnR(0 E ) are generated bv monomials, 
twcause cverything involved is ZS-gradcd. 
We descrit)e thc a-tight closure 0Ë of zero in E = (=E_«vra, k. x u. Let 
u.  _«v çq M. Thcn x '  0Ë if and only if .Equll q = 0 in Fo(E) or, equivalently, 
(qu+qP)ç((q- 1)w-«v)ç/1/= 0. for all q = pe. Dividing out by q, we cau rephrase 
this into the condition that (u + P) ç Int(w - «v) = 0, because -w/q  Int(-«v). 
Sincc this is equivalent to saying that Iut(u + P) ç (w - «v) = 0, it follows that 
.c '  (IË a if and onlv if Int(P)ç (w- u- er v) = la or, equivalently, if w- u ¢ Int(P). 
Now the equivalence of Colditions (1) and (2) follows immediately, because a 
monomial .r TM E R = k[«Vç1/] is in r(a) = Annn(0Ë ) if and only if x - ¢ 0Ë a. [] 

Example 4.9. Let ,ç = k[J'l .... , Xd] 1)e a polynomial ring and let R = S (') be the 
rth Veroncse sut)ring of S. We can easilv compute the idem r(a) = (a) associated 
to a monomial ideal a of/ï' as follows; cf. [How]. 
D,'c choose N and M to be the overlattice N = zd+ ;(1 ..... 1)Z and the 
sublattice /1I = {m  Z d I (m,n}  Z} of Z d, respectively, and let « be the first 
orthant in N = IR d. Then thc dual cone «v is also thc first orthant in/1I = IR d, 
and the ring /ï' = kit v ç MI is realized as it is as the rth Veronese subring of 
S = k[« v çZ d] = k[x .... ,x«]. In this setting, the vector vo (E I (for both R and 
S) dcfined in 4.7 is cqual to 1 = (1, 1 ..... 1). Also, for a monomial ideal a C_ R, the 
Newton polygons P(a) and P(aS) are equal to each other in/1I = IR a. Therefore, 
Theorem 4.8 relis us that a mononfial x TM in R (resp. S) is in r(a) (resp. "r(aS)) if 
and only if m, + 1  Int(P(a)) = [nt( P( aS) ), and, in particular, 

r(a) = r(aS) n 1¢ ; 

cf. Lemma 3.6. For exalnple, if a = m t is a power of t.he graded maximal ideal m of 
R, we bave r(m t) = m -{a-)/l = 

5. F-RATIONALITY OF PEES ALGEBRAS AND THE BEHAVIOR OF 3-(1) 

Throughout this section, we aSSulne that (R, m) is ail excellent Gorenstein local 
donmin of charactcristic p > 0, and that I is an m-prinmry ideal of R. Put d = 
dim R k 2. Let R(I) = R[It] denot.e the Rees algebra of I over R, and  = 
mR(I) + R(I)+, the unique homogeneous maxinml idem of R(I). We will denote 
by R'(I) = R[It, t -] and G(I) = R'(I)/t-R'(I) = (,>oI"/I + the ext.ende4 
Rees algebra and the associated graded ring of I, respectively. Also, let 
denote t.he graded canonical module of R(I), and let rr- Y = Proj R(I) -+ Spec R 
be the blowing-up with respect to I. 
The main purpose of this sect.ion is to describe cci( O in terres of r(I ') under 
the assumption that R(I) is F-rational. Actually, we prove the following theorem. 

Theorem 5.1. Let (R,m) be an excellent Gorenstein local domain of characteristic 
p > 0 with d= dimR_> 2. Let I be anm-primaryidealofR andJ its minimal 
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reductzon. Then r(I) Ç J  I d-l. If we assume that R(I) is F-rational. then 
r(I) = J  I d-1 and 
1 n l 
Discussion 5.2. The ai)ove theol'eln is motivated I)y Hyry's papers [Hyl], [Hy2]. For 
example, the description of 'RI) in Theorem 5.1 corresponds to the following fact 
used in [Hyl]: Let (R, m) be a regular local ring essentially of finite type over a field 
of chacterist ic zero, and let I be an idcal of R. SUl)l»OSe t hat Proj R(I) h rational 
singularities. The graded cauonical module of R(I) is then R(I) = > ff(I')- 
Actually, if R(I) is F-rational, then so is Y = Proj (I), whence Y is pseudo 
rational [Sml]. Therefore, if " has a resolution of singularities f X  Y, then 
H°(X, Iwx)  H°(I, I -) for every n  0. The left-hand side of this equality 
coincides with the multiplier idem ff(I ) via the isomorphislu wR  R as long 
as ff(I ) is defined. Moreover, [«R)],  H°(I ". I-), since R(I) is Coheu 
Macaulay: see e.g. [HHK]. In Im.rticular. we have r(I) = ff(I) in this case. 
Consequently, Theorem 5.1 claires that the F-rationality of R(I) gives a sucient 
condition for (I) = ff(I) to hold in any fixed positive characteristic.  
% obtain the following corollarv ffoto Theoreln 5.1 and [HWY1, Corollary 3.3]. 
Corollary 5.3. Suppose that (R.m) is a lwo-dimensional rational double point. 
Let I be an m-primary integrally closed ideal of R. and J its izial reduction. 
Then 7(I) ç J" I (= ff(I)). Also. R(I) .i.« F-ratio, al if and only if 7(1) = J" I. 
One can easily check the following example using the method developed in 
[HWY1, Section 3]. 
Example 5.4 (cf. [HWY1, Theorem 3.1]). (1) Let (R,m) be a twdimensional 
excellent Gorenstein F-rational locl ring (i.e., F-rational double point), and I an 
m-primary integrally closed idem of R. Then r(I) = J  I for any minimal reduction 
J of I. 
() Lt n = [[, . 4]/(«  +  + s), -h,- t: i 1 lg6ri«lly «loa la of 
chacteristic 2. Put m = (x, y, z)R and J = (y, z)R. Then R is a twdimensional 
rational double point, but hot F-rational. Also. we have: 
(a) R(m) is hot F-rational. 
(b) J* = m. In particular, J " J* = m. 
(c) (j[l)* = (g, z , «g). In particular, jDI  (j[l)* = (x, g, z). 
(d) (m) ç (x,. ) ç J(m) = m. 
In the following, we will prove Theorem 5.1. 
Lemma 5.5. Let (R, m) be a Gorenstein local ring of any characteristic. Also, 
let I be an m-primary ideal of R and put G(I) = o I/I+" the associated 
graded ring # I. Assume that [H(R(I)]0 = [H(R(I))]_ = 0 (e.g., R(I) is 
Cohen-Macaulay). Then R/['R)]  ([H(G(I))]_) v, where ( )v denotes the 
Matlis dual of R. 
Proof. Consider the following two standard exact sequences: 
0 R(I)+ R(I)  0. 

0  R(I)+(1)  R(I) --- G(I)  O. 
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From the first exaçt SC(luence , we have 
o [/-/,(R(l))]o H(R) [H+(R(I)+)]0 +' 
=   [H (n(I))]0 = 0, 
where th(, vanishing on thc right follows from a(R(I)) = -1. Since R is Gorenstein, 
V 
R =wn  (IIi(R)) v  ([H+'(R(I)+)]o) 0,, the other hand, the second exact 
sequence gives 
0 = [(())1_, + [(c())]_, + [H+I(()+)] + [+'(())l-, + 0. 
Dualizing thc ai)ove scqucnce, we gel 
d+l   0. 
This yiclds thc rcquired assertion.  

Proposition 5.6. Using [be saine notatio, as in Let, ma 5.5. assume further that 
R(I) is Chen-M««'aul«y. Tben [wR()] = .1 : I - for every utinimal reduction .1 
of I. 
Proqf. Let .ri, .r2 .... ,.rd be a minimal system of gencrators of .1. Pul G := G(I). 
Then it is well k«)wn lhat G(I) is Coh('n-Ma('aulay and the images in G of 
.rt ...... rot (G R(1)) fi)rm a regular Se(l)mnce ([GS]). S(,tting x := xit )nod 12 
for each i, wc lmve an exact sçqu(,nce 
î 
0  G(-1)  G  G/«;G  G(I/a',R)  0 
by [VV]. From this sequence, we get an exact sequence 
0 = H-'(G) H-'(G/xG) H(G)(-1)  H(G) O. 
-I + 1 = O. Bv repeating the above argument, we 
J +I d- 
[ (c)] [ (c/(" ")c] [ (c( ))] 
 H ..  .rd d- j 
H _ = , _ = H I/J = 
j  R/J " I a-. Combining this 
wi)h the previous lemma, we get 
[WR(l)], = A,mn ([H,(G)]-,) V = Annn ( J + I'-' ) v 
j =J" i d-. 

Proposition 5.7. Let (R. m) be an excelle)t Govenstein local domain of cbavac- 
teristic p > O, and let I be an m-primary ideal of R. Also, let J be a minimal 
reduction of I. Then we bave tbe followin9 statements. 
(1) 7"(1) C J" j.l Ç j. la-t. 
(2) IfR(I) is F-rational, then 7"(1) = J" j,1 = j. ia-a. 
Proof. Let xl,  .. , xa be a svstem, of generators of J and put j[tl := (xt, . , xta) 
for all integers l > 1. 
(1) One has r(I) C_ J  j.l C J  (J + I a-) = J " I a- by the definition of r(I) 
and Corollarv 2.8. 
To see (2), we lnay assunle that R(I) is Cohen-Macaulay. Then I a = JI d-a 
([GS]). Hence I al-I = ,lai-ai a-1 = (j[l]jal-a-I h- (xl '''xa)l-lR) I a-1 . Thus 



GENERALIZATION OF TIGHT CLOSURE AND MULTIPLIER IDEALS 3167 

j[t] + idl-1 _ j[q + (xl "" xd)l-lI d-l. In particular, for all 1 _> 1, we have J[q  
(J[q + I m-l) = (J[q  (Xl- "-x)/-1)  I a-1 = J" I a-. 
Now suppose that R(I) is F-ratiolml. Then since (j[q).l = j[q + idl-1 by 
Corollary 2.9, we have that J[q  (J[q)* = J  I d- for ail I _> 1. Hence r(I) = 
j. j.I = j. id-1, as required. [] 

Proof of Theorem 5.1. Note that R(I") is F-rational if R(I) is. Actually, it is a 
module-finite pure subring of R(I). Thus the required assertion immediately follows 
fronl Propositions 5.6 and 5.7. [] 

Proof of Corollary 5.3. Let I be an m-primary integrally closed ideal and J its 
minimal reduction. Then it is well kllOWll that I  = JI, and thus R(I) is Cohen 
Macaulay. 
It is enough to show that -(I) = J  I implies that R(I) is F-rational. Suppose 
that -(I) = J  I. Since (I) c_ j[t] . (j[q). c_ ,l [t]  (J[q+I "2t-) = .1  I, in general, 
we bave j[t] . (j[q). = ,l[q . (j[t] + i2/-1). This ilnplies that (j[t]). : j[q + i2-1 
for all ! _> 1, becmlse R/J [t] is an Artinian Gorellstein local ring. By [HWY1, 
Corollary 3.3(2)], we COllclude tllat R(I) is F-rational. [] 
In the rest of this sectioa, we will give sonm applications of Theorem 5.1. Before 
stating our results, let us recall the notion of o-invariant. Let I be an m-primary 
ideal of R, and put G := G(I) and 9 := mR(I) + R(I)+. Then the a-invariant 
a(G) of G is defined by et(G):: nlax{ G Z[ [Hd(G)]n ¢ 0}. Sec [GW] for details. 
Proposition 5.8. Let (R,m) be an excellent Gorenstein local domain of charac- 
teristic p > O. Let I be an m-primary ideal of R. Suppose that R(I) is F-rational 
and G := G(I) is Gorenstein. Then -(I ') : i,+«()+1 for all inte9ers n >_ 1. 
Proof. The F-rationality of R(I) implies that -(I ) = [wmt)], = H°(]; I'wr) for 
all n _> 1 sec Discussion 5.2. Also, since G is Cohen-lIacaulay, we bave 
(5.8.1) wa - G H°(]" I'-w )/H°(] ; I' w) ) 
>1 
and R = H°(Y,w) .) = .... HO(): I-a-lwy), where a = a(G) <_ -1; see e.g. [Hy2, 
Theorem 2.2]. On the other hand, since G is Gorenstein, we bave 
(5.8.2) 'c  C(a) = ( I'+«/I '+°+'. 
Comparing (5.8.1) with (5.8.2), one con easily sec that -(I ') = i+«+a bv induction 
onn> 1. [] 
Corollary 5.9. Let (R, m) be an (excellent) regular local ring of characteristic 
p > O. Then -(m ') = m -d+l for ail n >_ 1. 
Proof. Suppose that R is a regular local ring. Then R(m) is F-rational and G(m)  
k[X1, ...,Xd] is Gorenstein with a(G(m)) = -d. Hence we con apply the above 
proposition. [] 
Remark 5.10. Corollary 5.9 is a generalization of the implication (1) == (2) in 
Theorem 2.15. This also follows from Theorem 4.8. 

Let J c_ I be ideals of R. Recall that the coefficient ideal of I relative to J, 
denoted by a(I, J), is defined to be the largest idem a of R for which Ia = Ja. 
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Re'ma.'rk 5.11. In [Hy2], llyry proved that if _/7 is a Gol'enst(nl local ring and R(I) 
has rational singulal'iti(,s, t hcn J([d- 1) = a( I, ,1). In fact, a silnilar result follows 
from Theol'eln 5.1 and [Hy2, ThcolCln 3.4]" Let (R. m) he an excellent Gorenstein 
local domain of «haractel'istic 1» > I1. Let I be an m-prilnary ideal of R, and J its 
lninilnal l'(,duction. If (I) is F-rational, thell r(I d-) = H°0 . la-wr) = a(l, J). 
In parti«ular, if, in addition, I2= ,1I, then r(I 't-) = .I" I. 
In the test of this section, we direct, our att('ntion to the idem r(m). Let (R, m) 
ha an excellent (lOl'(mstein F-rational local domain of characteristic p > 0. Then 
r(m)  m, that is, r(m) = m or r(m) = R. FO)l" example, if H is a regular local ring 
with diln H OE 2, then r(m) = l. gh»re gcn¢,rally, we have the following proposition. 
Proposition 5.12. Let (.m) b« a e.rcelle# Goresteb local domain of char- 
«u't«ri.stic p > Il witl d = (lira I  2. Stt»t»oe that there exists an m-primary 
id«(! I sm'l ttat R(I) is F-rational 
r(m) = B. 
Pvof. ]{y Vill(' «)f [IIWY1, ('or«»llaly 2.13],  is F-rational. 
Bv The(»rem 5.1, w(  havc r(I) = .1 " I'-  fi)r any lninilnal rcduction ,] of I. Since 
R(I) is (:uhen Macaula) with a(C(l)) ¢ -l, we havc that I -] = JI d-2 ç J. 
llence r(l) = R. In particular, r(m) = 
In view of the ai»ove proposition it is narreal to ask the following question. 
Question 5.13. Let (R. m) bc an exc(qlent Gorcnstcin F-ra.tional local dolnain of 
('haract(,risti(" p > 0 with r(m) = R and dira R  2. XVhen is R(m) F-rational then? 
hi Ihe case of two-dimensional local rings, r(m) = R ilnplies that R is regu- 
lar. Then R(m) is Gorelistein and F-rational with ((G(m)) = -2. As for three- 
dimensional local rings, we have t.hc following answer to the ahove question. 
Proposition 5.14. Let (R. m) be a three-dime.sioml e'cellet Gorestei local 
rit 9 tbat is ot reg'ttlar. The tbe followi 9 coditios are equivalent. 
(1) r(m) = B. tiret is, /*m = [ holds for every ideal I 4 R. 
(2) ,]* = ,1 lolds fo'r some parameter ideal ,1 of R. 
(3) R'(m) is F-ratioml, and m  = Jm for some miimal reductio J of m. 
(4) R(m) is Gorest«b a,d F-ratioml. 
(5) r(m ) = m'-] twld.s for ail ite9crs  k 1. 
Pro@ (1)  (2) and (5)  (1) are trivial. (4)  (5) follows from Proposition 5.8. 
in«e «(G(m)) = -2 
To see that (2)  (3), wc may aSsulne that ,1 is a llfillinlal reducIion of m: sec 
Discussion 1.14. Bv Corollarv 2.8. we have m 2 C J, and thus m 2 = Jm. Also, 
R'(m) is K-rational by [HWY1, Corollary 4.5]. 
(3)  (4): Note that a Gorenstein local ring having minilnal multiplicity is a 
hypersui'Nce with nmltiplicity at most 2. Thus R and G(m) are hypersurNces, and 
a(a(m)) = 1- dimR = -2. Hence 
is Gorenstein and F-rational. R(m) is F-regular. 
Discussion 5.15. We tan generalize the equivalence of (2) and (3) in Proposition 
5.14 as follows; sec also Theoreln 2.15. 
Let (R, m) be an excellent equidimensional reduced local ring of characteristic 
p > 0. Then the following conditions are equivalent. 
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(1) R'(m) is F-ratiollal, and m 2 = dru for some minimal reduction J of m. 
(2) J*'-= = ,1 holds for every (or equivalently, SOlne) parameter idem J of A. 
If, in addition, R is Gorenstein, then the following condition is also equivalent 
to the above conditions: 
(3) T(m d-2) ---= /. 

Example 5.16 (cf. [HWY2, Proposition 3.12], [HWY1, Sect. 5]). Let (R,m) be an 
excellent three-dimensional Gorenstein normal local domain of characteristic p > 0. 
If R admits a non-zero-divisor f E m such that R/fR is F-rational, then r(m) = R. 
For example, let R = k[[x,y,z,w]]/(x 2 + ya + z b + u,«), v«here k is a field of 
characteristic p > 0, and a, b, c are integers with 2 < a < b < c « p. If 1/2 + 1la + 
1lb > 1, then r(m) = R. Otherwise, r(m) = m. 
Rernark 5.1ï. (1) If R is a three-dimensiona[ regular local ring, then r(m -) = R 
(and thus 7-(m) = R) and R(m) is F-rati«mal. But R(m) is hot Gol'enstein. and 
7"(m'") : m n-2 for all n, _> 1. 
(2) We bave no examples of a Gol'elsteiu local ring R fi)r which r(m) = R but 
R(m) is hot F-rational. 
Discussion 5.18. Let (R, m) be a complete regular local ring of characteristic p > 0 
with d = dira R > 2, and let I be an m-primary ideal of R. Then we expect that 
7-(1) D I. 
For example, this is true if R(I) is F-rational. We sketch a proof here. Suppose 
that 7-(1) = I. Then 7-(I") C_ 7-(I)' = I' for ail n _> 1, by the subadditivity 
(Theorem 4.5). On the other hand, since / is F-regular, we have 7-(I') _ I' in 
general. Also, by Theorem 5.1, we have rt) = (>1 I' = R(I)+. In particular, 
since R(I)/wrt()  R is regular, so is R(I). (Note: Recently, S. Goto et. al. proved 
a nore general result.) But this is impossible, because dira R _> 2. Hence 7-(I) D I, 
as required. 
As for nmltiplier ideals, the authors are inforlned of the following result bv N.- 
i. Watanabe: Let R be a regular local ring essentially of finite type over a field of 
characteristic zero. Then ,7(1) D I for any m-primary idem I of R. [] 

6. RATIONAL COEFFICIENTS 
Recently, the theory of multiplier ideals v«ith "rational coefficients'" has been 
developed and applied successfully to various problems in algebraic geometry and 
comlnutative algebra ([ELS], [La]). This lnotivates us to extend the notions of a- 
tight closure and the ideal 7-(a) to those with "rational coefficients". In this last 
section we nlake a few remarks on rational coefficients mld address the results that 
generalize in this setting. 
Definition 6.1. Let a be an ideal of a Noetherian ring R of characteristic p > 0 
such that a ç R ° -¢ 3, and let N C_ M be R-modules. Given a rational number 
7-.a t 
t > 0, the t. a-tight closure r.t., (or, a-tight closre 't sec Relnark 6.2 (1) 
below) of N in M is defined t.o be the submodule of M consisting of ail elements 
z E M for which there exists c G R ° such that 

czqI:I [tq] _ ]viÏ ] 
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fol" ail q = pe» O, where. [tq] denotes the lea.st integer that is greater than or equal 
to tq. 
R«mark 6.2 (1) Dcfinition 6.1 docs hot, change if we replace "czO [to C [" 
(roundiug up tq] bv "czO [to Ç [" (rounding down tq),  long s a  R   0. 
  M 
This is because the difference of [tq] and [tqJ as the exponents of a is "absorbed'" 
by the terre c  o. Similm'ly, it is easy to see that t- a-tight closure is the sa.me 
tn-a-tight chsure fi)r every nonnegative integer n; cf. the proof of [HXV. Proposition 
2.6]. This being so, it is prefeable to sav "a*-tight çosure" rather than "t  a-tight 
closure." In the sequel, we alwoEvs nse "exponential notation" in this ruminer and 
denote the at-tight closure of Nin [ by r*' 
(2) The above fornmlation extends to several rational coefficients (or. several 
rational exponents). Namely, given ideals at, .... a,, ç R with a,  R   0 and 
rational mnnbers t,...,t  0, if ri = t,i for nonnegative t G Q and ni  Z with 
i = 1,...,r, we can define aï..- at-tight closure to be (aï--. a't-tight closure. 
If N is a submodule of an R-module 1, thcn an element z 
tight closure 
"'M of N in .1 if and onlv if there exists c  R  such that 
 [tq] ..atq] ÇN[Ï] for allq p»0. 
CZI  .  
(3) k also have an a.nalogous notion of A-tight closure for a pair (R. A) of a 
uormal ring R and a Q-kil divisor A on Y = Spec 
and A  _ _ _ 
= i= t  divr(s'i) for xi 6 R and 0 < t  Q with 1 < i < n, then 
aï .-- at,-tight closure is the smne as A-t.ight «losure. 
Definition 6.3. Let  be an idem of a Noetherian ring R of characteristic p > 0 
such that a  R  ¢ 0, and let t  0 be a rational retomber. b say that an element 
c  R  is an af-test element if czq [tq] 
Some results for a-tight closure generalize to those for at-tight closure without 
essential change of proofs. However, we must be careful about the difference of 
romM-up and round-down when speaking of at-test elements. As a matter of fact. 
the following theorem is proved similarlv to Theorem 1.7 (1), but the proof does 
hot work if we replace [tq] by [tqJ in Definition 6.3. 
Theorem 6.4. Let R be an F-finite reduced n9 of charactestic p > 0 and let 
c  R  be an element su, ch that the lo«alized n9 R« is stnglg F-regular. Then 
some power c  of c is an a t-test element for all ideals a ç R with a  R  ¢ 0 and 
all rational numbers t > O. 
 can define the ideal v(a ) in a similar way to Proposition-Definition 1.9. Also, 
Theorem 1.13 generalizes to the ce of v(a t) with the saine proof, k summarize 
the results for excellent reduced local rings in the following. 
Definition-Theorem 6.5. Let ( R. m) be an ecellent reduced local tin9 of charac- 
teristic p > 0 with E : En(Rira). and let 
Given a rational number t  O, we define the ideal r(a t) ç R by 

= = Ann(O^, ) = N (I" 

AI ^tÇE tçR 
where iI r,uns thro'ugh all finitel9 9enerated R-modules (resp. finitel9 9enerated R- 
subm, odules of E) in the second terre (resp. the third terre,), and I z'uns thro.ugh all 
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ideals of R. Moreover. if R is normal and Q-Gorenstein, then 
r(a') = Aml(OË'). 
tri 
Remark 6.6. We can define the idem T(aï ..-a, ) with several rational coefficients 
by replacing at-tight closure in 6.5 by aî 1 --- a/-tight closure as defined in Remark 
6.2 (2). Sec Theorem 6.10 (2). 
Proposition 1.15 also generalizes without essential change of the proof, but we 
cannot replace the round-up [tq] by the round-down [tq] in the following. 
Proposition 6.7. Let (R,m) be a d-dimensional excellent normal local ring of 
characteristic p > O. a C_ R an idem such that a ç R ° ¢ O, and let t >_ 0 be a 
n*«' is the unique maximal proper submodule N witb 
rational number. Then "'H() 
respect to the property 
artq] F«(]V) c_ N for all q = p, 
where F¢: H(R)-- Hd(R) is the e-rimes iterated Frobenius induced on Hd(R). 
Now we generalize Theorcm 3.4, which is the lnain theorem of Section 3. to the 
case of rational co('fficients. 

Hd(R) 
where E is the closed fiber 
Here we note that the 
x(-LtzJ) = x([-tz]), 
if R is quasi-Gorenstein. 
YI *OEt _ 
The inclusion ,Hd (R) D 
characteristic p > 0: Just 

Theorem 6.8. Let t >_ (1 be a fixed rational nu-tuber. R a normal Q-Gorenstein local 
ring essentially of finite type over a field, and let a be a nonzero zdeal. Assume that 
a Ç R is reduced from characteristic zero to characteristic p » O. together with a 
log resolution f: X - Y = SI)eCR of the ideal a such that aOx = Ox(-Z). Then 
r(a t) : H°(X. Ox([tçx/y - tZ])). 
Sketch of the proof. This is also proved in a similar way to Theorem 3.4. So we 
just indicate the points where some modification is needed in the following. First, 
we note that Lemma 3.6 holds for rational coefficients without changing the proof, 
i.e., r((aS) t) ç R = r(a t) under the assumption of Lelmna 3.6, and that Lemma 3.7 
is already proved for rational coefficients. Hence we can use a canonical covering 
of R to reduce the proof of Theorem 6.8 to the quasi-Gorenstein case. Then it is 
sufficient to prove the following generalization of Theorem 3.9. 
Theorem 6.9. Let t >_ 0 be a fixed rational number. ( R. m) a d-dimensional normal 
local ring essentially of finite type over a fiel& and let a be a nonzero ideal. Assume 
that a C R is reduced from characteristic zero fo characteristic p » O. together with 
a log resolution f- X --, I" = Spec R of a such that aOx = Ox(-Z). Then 
= Ker (Hdm(R) HÊ(Ox(tZ)) ) , 
of f and  is an edge map as in Proposition 3.8. 
canonical dual of the sheaf Ox(tZ) = Ox([tZ]) is 
which is isomorphic to Ox ([Içx/n - tZ]) via wh - R 

Ker6 of the above theorem holds true in arbitrarv fixed 
take an element c in the proof of Proposition 3.8 from 
¢1 [tq] _ H°(X, Ox(-tqZ)) instead of a q = H°(X, Ox(-qZ)), which gives rise to 
a map cF «" HÊ(Ox(tZ)) -- HÊ(Ox(tZ)). Then one secs that a [tq]F(Ker() c_ 
Ker( for all q = p, and Proposition 6.7 applies. 
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a*«' C Ker 5, we choose, in characteristic zero 
To prove the reverse iuclusion "'H(,) -- 
before rcducing to characteristic p >> 0, a nonzero element c  a such that Re 
is regular and a log resolution f: X  Spec R of the idem ca, as in the proof of 
Theorcm 3.9. Thon choose an f-ample f-exceptional Q-Cartier divisor D and a 
sufficientlv small e > 0 so that [2 + edivx(c)] = [tZJ, where 2 = tZ - D. 
then move to reduction modulo p » 0 and let m = r + 2s + [n0t], keeping the 
integers r, s, no just the saine as in the proof of Theorem 3.9, i.e., the idem a is 
generated bv r elements, c  is a usual test element and also an at-test element, and 
 = >0 H"(X,x(-[n2J)) is generated in degree  n0 as a graded module 
over  = ,,>0 H°( X' Ox (-t2)). Now, arguing as before, we obtain the required 
 *at 
inclusion "H() C KcrS. 
Finally, we noie that the results in Section 4 also generalize to ratioiml coeffi- 
cients, with lhe saine l,roof: sec [DE], [How], [Lai for the corresponding results for 
multiplier ideals. 
Theorelll 6.10. Let t, t' be a9 nomte9ative rational numbers. 
(1) (Resti'içtiou theorem): U, td«r the assumption of Theorem 4.1 we bave 
r((as")') g r(a')'. 
(2) (Subadditivity in rcgular local rings; cf. emark 6.6)" Under the assump- 
tion of Theom 4.5 we bave 
(3) Uder the assumpon of Theom .8. le  ç R be  orc ideal. Then 
ç()  (). and it 'is also  toric idea! 9eeraed @ onomials z  with 
m  M such that 
m +., 6 hat(t- 
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SESHADRI CONSTANTS ON JACOBIAN OF CURVES 

.IIAN KONG 

ABSTRACT. çVe coznpute the Seshadri constants on the Jacobian of hyper- 
elliptic curves, as well as of curves with genus three and four. For higher genus 
curves we conclude that if the Seshadri constants of thcir Jacobian are less 
than 2, then the curves must be hyperelliptic. 

1. INTRODUCTION AND STATEMENT OF RESULTS 
Let X be a smooth «omplex projective variety. Let. L be an ample line lmndle. 
Let p  X be a. point. Define he Seshadri constant of L at p to be the real mmber 
e(L,p):=ilff nliçÇ ] p CCX . 
Here the infimum is taken over all reduced curves C passing through p, and multçC 
is the lnultiplicity of C at p. Another equivalent definition is 
e(L,p) = sup{e If*L-cE is nef}, 
where f : BlvX  X is the blow-up of X at p and E is the exceptional divisor. 
The Seshadri constant indicates how N.r the ample divisor is ff'oto the boundary 
of the ample cone near point p, and thus lneasures positivity, or ampleness locally. 
The study of Seshadri constants h drawn increasing interest during recent years. 
For properties of Seshadri constants sec [1] and [6]. 
In the case of abelian variet.ies, it is known that a general element in the moduli 
space of principally polarized abelian vm'ieties of dimension g has Seshadri constant 
very close to its maximum upper bound ([4]). On the other hand, there are some 
special abelian varieties, namely .lacobian, which have relatively small Seshadri 
constants. çVe will discuss some cases in this paper. 
Let C be a smooth projective algebraic curve over C with genus 9 = 9(C) k 2. 
Denote O to be the theta divisor of J(C), its Jacobian (recall J(C) = Pic°(C)). 
Since abelian varieties are homogeneous spaces, we can define 
for any p  J(C). 
2g 
It is known that 1 < e N , and if C is hyperelliptic, then e   ([4], [5]). In 
4 
particular, if 9 = 2, then C is hyperelliptic and it is known that e = g ([6]). The 
problem becomes very interesting even when 9 = 3. The point here is to sec if the 
Seshadri constants tan be their maximum, i.e., , thus most of the time irrational, 
or always less than their maximum - and thus more likely rational. While all the 
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existing exalnples suggest the latter, we investigate this prot)leln in detail, lnainlv 
by looking al the cases when e _< 2. 
Ollr main result is the following theoreln: 
Theorem 1.1. Assume the Picard number of J(C) is one. Then: 
(1) If C 
9+1 " 
(2) If9 = 3 ad C 
(3) If9 = 4 ad C is net hyperelliptic, then  = 2. 
(4) If g >_ 5 and C is ot hyperelliptic, then e >_ 2. 
Part (4) of the theorem can t)e restatcd as: 
Corollary 1.2. If g >_ 5 aud  < "2, th«v C is hyperelliptic. 
Remark 1.3. (1) Foi the case of calculati(m (Jn the Neron-Severi group of the sym- 
metl'ic pro(hwt (72, we need that il is generated t)y a fit)er and the diagonal, i.e., its 
Picard lmllll)er is '2. That is truc if C is of gelwral m(J(hlli. V% need this condition 
through(mt this pa.l)er. But this restlicti(Jn, h(Jwever, seems te be liot essential. 
(2) We ('ail also locale ail the special curves that give relativel,v small ratios in 
case () t (3). 

2. PROOF OF TtlEOREM" ||YPERELLIPTIC CASE 
The following ot)servatioll, while straightforward, points out where we want to 
find special curves that give the exact value of thc Scshadri coustants. 
Lemma 2.1. lf C' is an irreducible curve in J(C) suc]) that c'.o < 9 then for 
multoC  -- -. 
any divisor D with D =- kO ad mult0D > 2k, we bave C' C D. 
If C is h,vpcrelliptic, then the case of k = 1 in Leumla 2.1 reads D = O and 
lnult0D 2 2, which we denote as (*). For d 2 '2, let Cd be the d-fold SVlmnetric 
product of C (the set of effective divisors of degree d on C). 
Proposition 2.2. Let u " Cd -- J(C) bc tbe Abel-Jacobi map. Tbcn 
ll  : ,,(c). 

(.) 
Pro@ Let L be a hyperelliptic line bundle Oll C. Let Po be a ramification point of 
the 9; se L = Oc(2p0). We fix a translati(Jn ofthe Abel-Jacobi lnap 
by sending l" Ç Ca te Y - deg()'). Po  ,I(C), and for simplicit,v we ignore the 
Po part for representation of points in ,I(C) in our proof. Recall that 
Cg_i, l  -- Y + L lnaps Cg_a biratiolmlly and surjectivelv te Sing((0) iii this case 
([31). 
For an3" l"  Cg-3, define D)- = (O - Y. Il translates Y + 2p0  Sing(O) to 
0  D)-. Thus D)- =- (-) and nmlt0D 2 2. Il suffices to show that Ç)D)- = u(C2). 
Il is obvious that t(C2) C Ç)D)-, since fol" anv point (p, q)  C2 we can rewrite 
it as (p + q + )') - )"  D)- for an)" )"  Cg-3. 
On the other side, an)" points in Ç)D)- can be represented as D - Y for some 
D Ç Cg-1 and Y  Cg-3. Also, since il is in the intersection, for any F Ç Cg-3, 
there exists E Ç Cg-1 such that D - )" = E - F, i.e., D - l" + F is (equivalent to) 
an effective divisor for any F. Wc claire D - :t" itself must be effective, and since il 
has degree 2. it is in u(C2). 
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Pick a representation of D - t such that t" contains no ramification point of 
91- First assume that D contains no hyperelliptic pair. ff D - t" is hot effective, 
pickp 6 " but p  D, and let. L = Oc(p+p'). C, hoose F = Y-p+p'. Then 
D- Y+ F = D-p + p'. But the linear system I D-p + P'I is empty, since otherwise 
it must cont.ain a nmltiple of hypcrclliptic pairs and base points, which will lead to 
pED. 
If D bas some hyperelliptic pairs, then cancel as many points in D-Y as possible 
until either D - Y is effective or D rms ont of hyperelliptic pairs and reduce to a 
similar sit«m.tion in the first case. [] 
If C is hyperelliptic, and rk(NS(C2)) = 2, then A'S(C2) is generated by F and 
A, t.he image of a fiber and the diagonal ff'oto tbe natural map C x C - C2. 
There is a rational curve, call it pi, that consists of hyperelliptic pairs {(p,q)  
C2 [ Oc(p+q) = L}. Also denote u*(6)) still as O. We list the nmnerical properties 
of NS(C2) below. 
Lemma 2.3. I/ïth notation as above, u,e bave: 
(1) O = (g + 1)F- 1 pi 1 
A ad = 2F- A. 
(2) F 2=I,F-A=2,A 2 =4-4g. 
The Abel-Jacobi map u " C2 -- J(C) contracts P and is isolnorphic outside 
of P. Now let C" be an irreducible curve in C2 not contracted by u and let 
C' = u( C"). Then 
C" " (-) C" 
C"  p1 mult0C " 
So out theorem in the hyperelliptic case follows from the following proposition. 
Proposition 2.4. Among all irreducible curves in C2 hot contracted by u. A is 
the only curve with minimum ratio ,_x. _ 2_9_ 
A.P -- g+l" 
Pro@ Since A   = 49 and A. p1 = 29 + 2, we have A.O 49 _ 29 
A.P -- 29-t-2 -- 9+1" 
Let Co = oF + bA C C2 be an irreducible curve not contracted bv u. Then 
Co" p1 = a q- (29 + 2)b _> 0 and Co '  = (a + 4b)g > 0. Then 
A .  (a + 4b)g 29 
-- >--  a>0. 
A  P a + (29 + 2)b 9 + 1 
But if a _< 0, then we must have b > 0 since Co"  = (a + 4b)9 > 0. Now we have 
C0  A = 2a + b(4 - 49) < 0. Since both C0 and A are irreducible, C0 = A. [] 
Remark 2.5. A little more det.ailed calculation shows that A is actually the only 
curve whose corresponding ratio is less thon two. 
3. PROOF OF THE THEOREM: NON-HYPERELLIPTIC CASE 
If Cis non-hyperelliptic, then choose the case k = 2 in Lenmla 2.1 which reads 
D = 2 and nmlt0D >_ 4. Denote this linear system as [23100. So we look at the 
base lotus of 123100. Here we need the following result of Welters. 
Proposition 3.1 (Velters [7]). Bs(12100) -- A(C × C). Here A" C × C - J(C), 
A(p, q) = p- q. is the difference map. 
Remark 3.2. Welters' theorem is truc for all curves with 9 = 3 or 9 > 5. For 9 = 4 
the base lotus has two more isolated points, which will hot affect our proof since 
we are looking at curves inside the base lotus. 
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In this case we look at the Neron-Severi group in (' × C, It is geuerated by fibers 
F, F 2 and the diagona.1 . e list. their mmwrical l)rop('rties below. 
Lemnm a.a. ll'it notation as above, we bave: 
(1) *(-) = (9-- 1)(& + ) + . 
(2) F=0. F.&=F.=I. Ae=2-2v.i=I,2. 
Sinçe C is non-hyperelliI»tic, the difference map A contracts the diagonal  to 
0  .I{C) and is isomorphic outside . This enables us,  similarlv in the hyper- 
elliptic case, to shift lhe computation ffoto the ratio c.o in .I(C to the ratio 
multoC 
of thc intersection nmnbers in the Neron-Severi group, whi«h are well understood. 
Specifically, let C" be a.n irredu«ibh' «urve in (' x C hot contracted by A and let 
C' = u(C'). Then 
C" . A * @ C' . @ 
C"   multoC" 
So out theorem in lhe non-hylwrelliptic cse follows from the following proposi- 
 ion. 

Proposition 3.4. ll'ith notation as above: 
C".A*O 
(1) If g = 3, Otc minimm ratio c". is  for curves in C x C, and is 
a«hieved by one «urve. 
(2) If9 = 4, the minimum ratio c". is 2 for curves in C'< C, and is achieved 
by more thon o'ne cure, e. 
(3) If g > 5, then c".x'o > 9 for ail curves in C x C hot contracted by A. 
Pro@ (1) 9 = 3: In this case, the canonical systcm embeds Cas a plane quartic. 
Let Oc(1) be its hyperplane section. Consider the curve 
C0 = {(p,q) I Oc(p+ q + 2r) = Oc(1) for some r 
Write C0 = oF + bA. Since C0 - A = 56 (twice the number of bitangents) and 
C0- F = 10 (degree of the ramification divisor of the dual curve's 9à), we can solve 
for a and b and get C0 = 16F - 3A. C0 is irreducible since it is isomorphic to C 
viap+qr. Pullingit backtoCxCweget acurve C = 16(F +F2)+6A. 
Now 
c2. a*o [1( + v) - 61. [2(Vl + ) + 
C-A [16(Ft + F2)- 6]-A - 56 7 
To claim that  is the minimum ratio, let C = aF +b +C5 be anv 
irreducible curve in C x C hot contracted bv A. If C" ¢ C, then C"  C = 
10(a + b) + 56c k 0. So if c OE 0, then 
12 
C"-*O_ 3(a+b) >3>. 
C . a+b-4c - 7 

If c < 0, then 

C .A*O 3(a+b) > 3(a+b) 7 12 
C".A a+b-4c- a+b+(a+b) 4 7 

This shows that the only curve that achieves t.he minimum ratio  is C'o'. 
(2) g = 4: In this case C has two gâ's. Let L be one gà. Consider the curve 
C0 = {(p,q) I I L - P - ql > 0} c Ce Since C0- F = 2 and C'o- A = 12 (degree 
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of ramification divisor of L), we find that Co = 3F - 1 
A. Lift to C x C to get 
c;, 
= 3(F1 + F2) - A. A silnilar calculation as above shows that 
 A*O 24 
C e .A 12 
and it is t.he lllillillllllll ratio that tan be achieved on C x C. 
Note that iii this case there is allother cmve (froln the other g) that gives the 
nlillinllllll ratiO The reason is that in this case Cô = 0, while in the case of g = 3 
we have Cô < 0 (thus Ulfique). 
(3) g >_ 5: Assume C has a girl (d >_ 3), call it L. As in (2), consider the curve 
Co = {(p,q) I IL -P-ql > 1)} c C2. Tllen C0- F = d- 1 and Co-/k = 2d + 2g- 2 
(degree of ralnification divisor of L). Thus Co = dF - }A. Lifting to C x C we get 
C = d(F + F) - A. Now first we have 
C o  A*O 
- >2. 
CD'.A d+g-1 
Secondly. for allV irreducible C" = aFt + bF + cA C C x CllOt contracted 1)y A, 
either C"  C' o' < O. or 
(7 .&*(9 d + g- 1 
> >2 ifd_<g- 1. 
C".A - d - 
Since the Brill-Noether imlnber for g is non-lmgative if d > g+2 both ratios above 
C"-A' < 2, then ail the clrves Co nust be 
are at least 2. If the lllillilllUlll ratio c".x 
reducible and contain ai1 irreducible colnponellt C1 whose lift to C x Cgives a 
small ratio. It is easy to sec that C < 0. thus lulique iii C2. This is certainly 
ilnpossible. (For example, if there are two different 9  for some d, then to have a 
d 
COlllnloil Colnponent for corresponding Co C C, Olle coordilmte has to be a base 
point of g Thus it is a linear Colnbination of fibers, and since the component is 
d" 
irreducible, it is a fiber. But for a fiber, the corresponding ratio is g > 2.) 
Note that the minimmn ratio exists and tan be achieved if dg <  
d+9--1 -- d 
which is equivalent t,o dg < X/, or d <  + 1 [] 
d+g-1 -- --  

4. OTHE1R RELATED PROBLEMS OF SESHAD1RI CONSTANTS 

For non-hyperelliptic cases when g _> 5. to find the Seshadri constants, the first 
step is to look at the curves in C2. It is related to the problem whether the cone 
of effective curves of C2 is closed. If it is, the curve from the boundarv will give a 
better upper bound of e() which is less than . In all special ces that we have 
discuss (hyperelliptic, snmll genus, curves with g for small d), the cone is close& 
For the general case there is solne indication that the curve from one boundary 
(the other one being the diagonal), if close& will give the ratio  where (p. q) is 
p 
the primitive solution of Pell's equation x  - gy2 = 1, if g is hot a square The 
following example gives some indication that it could be truc. 
Example 4.1. If C is a plane quintic (i.e., genus is 6), consider the cnrve Co = 
{(p,q)[]Oc(1)-p-q-2r > 0} C Ce Then Co is irreducible and Co = 50F- 7. 
Since any curve C C2 satisfies Co  0, a. calculation shows that on C x C, 
c" 'o > ç holds for all irreducible curves (except A). Note that the bound ç is 



3181) J1AN KONG 

wha.t the conjecture gives. Also note that one expects small values fi»r plane curves 
that are Sl)ecial in tlw moduli of curves, so general curves of genus 6 must also 
satisfy lhat 
For 9  5, if e ( 2, then it follows tiret CE is hyperelliptic. In ail ces, hyperel- 
liptic curves give us the smallest Seshadri const.ams. From the known result ([2]) 
of Bs(2(-)[00) in dimension 4, it is easy to see that: 
If A is an indpcomposablp principally polarized abelian variety of dimension 4 
and e((-)) < 2, then A is the 3acobian of a hyperelliptic curve Cof genus 4. 
lt, is very rea.sonalde to ask the saine question for any genus and seems like it 
«ouhl be lrue. 
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ON THE CLIFFORD ALGEBRA OF A BINARY FORM 

RAJESt! S. KULKARN1 

ABSTRACT. The Clifford algebra C i of a binary form f of degree dis the k- 
algebra k{x,y}/1, where I is the ideal generated by {(x-F y)d _ f(c,/3) I 
,/3 E k}. C i has a natural homomorphic image A i that is a rank d 2 Azu- 
moEva algebra over its center. We prove that the center is isomorphic to the 
:d+g--I 
coordinate ring of the complement of an explicit )-divisor in P'C/k , where 
C is the curve {w d - f(u,v)) and g is the genus of C. 

1. INTRODUCTION 

Let f be a forlll of degree d iu n variables over a field k. Then the Clifford algebra 
C.f is the k-algebra k{Xl .... ,x,}/I where k{xl ...... r,} is the free associative 
algebra in n variables and 1 is the ideal geuerated by {(CqXl + ... + ax) a - 
f(al ..... c,) } oq,...,c,  k}. If d = 2. this is the classical Clifford algebra of 
a quadratic form. If d > 2, theu this is solnetinles called the generalized Clifford 
algebra and has been studied by various authors, iucludiug Roby [21], Revoy [20], 
and Childs [3]. 
The first case of higher degree (that i, d = 3, n -= 2) vas studied bv Haile in [8] 
and [9]. He showed that the Clifford algebra of a binary cubic form over a field k, 
with characteristic  2, 3. is Azunmva with center the affine coordinate riug of an 
elliptic curve. Further, this elliptic curve is the Jacobian of the projective curve 
given by the equation (W 3 -- f(u, V)). 
The case of d > 3 disploEvs different behavior. Namely, using results of [20], it 
is easy to see that C i contains a free algebra on two variables. Iu particular, C i 
is hot finitely generated over its ceuter (as a nlodule) and hence is uot Azmnaya. 
However, it has been shown ([10]) that the dimension of any representatiou is 
divisible bv d. Furthenuore, Ci = Cy/(Ç ker q) is Azumaya over it.s center, where 
the intersectiou is takeu over ail the keruels of dimensiou d represeutations. The 
question of describiug the center is thus interesting from this point of view. 
The main tool in understanding representations of the Clifford algebra (for d > 3) 
was iutroduced by Van deu Bergh in [22]. While previous results about represeuta- 
tions ([8], [10]) were global iu nature, this idea gave a local result. To uuderstand 
the statenlent we introduce some notatiou. Denote by C, the projective curve given 
by the equation (w a - J'(u,v)). Let g be the genus of this degree d curve. As- 
sume that k is algebraically closed of characteristic 0 and that the forln f has no 
repeated factors over k. By au rd-dimensioual representation of C'y, we mean a 
homomorphisnl from C i to Alrd(k), the rd × val matrices over k. Then the result 
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([22], Prot)ositiol 1 in Section 1 an(l Lemma 2 in Section 2.2) states that there is a 
one-to-ole çorl'eSl)OlUl(,nce i)'tween equivalençe classes of 7"d-dilnensi(»llal represen- 
tations of ('f ail(! iSOlllOrl)hisln classes of vector i)lin(l|es " on C of rank r and degree 
(d + g - 1) such that H°(C,g(-1)) = I). It shonld be enq)hasized that this result 
is constructive iii llatlll'e. Nalncl.v, for a vector blll(lle that satisfies the conditions 
of the theol-enl, we can construct an explicit representation of C I (on the global 
sections of ,5"), and vice versa. 
This result snggests that the çellter of C I should be the coordinate ring of an 
at:tïne open set iii thc (translated) ,Iaçobial of thc çurve Ç'. In fa.ct, a closcr inspec- 
;Ad+g-- 1 
tion rcvcals that the open set is the complem(,nt of a (0-divisor in Pa-c/ . The 
results of [8] «md [9] pertaining to llle case of binary cubic forlns give credence to 
this i)elief. In this l)aper, we prove this resul! ramer the assumt)tions that k is an 
infinite tiehl, f has ne) rel)eate(l factors over an algei)raic closure of k and char(k) 
(loes nt dividc d. 
First considcr the case when C has a k-rational 1)oint. For this case, ve briefly 
(iesclibe the strategy to c«mstru('t the re(lnired isolnOrl)hism. Let Zk denote the 
('enter of Çf and .4t. (If'ne»t(' the c)oldillatering of the COlllplelllellt of a (-)-divisor 
 . d-t-g- 1 
in V1CE/k ,VP C()lllpli[e the graded module associated to the direct image of the 
mfiversal imndle Ululer th(, l)rojectiol C xk Spec .4 -- ]P. x b'l)ec A,.. In fact, it 
turns out that this (lirect ilnage is isolnorl)hic to thc pull-back of the direct image 
of the mfiversal bundle under the t)rojcction on the second factor. This allows us 
to (lefine a homonlorphisln ç from C I to End..P0. where Po is the Ak-module of 
global sections of the direct image of the universal line bundle. This morphism 
bas the important property that any d-dimensional representation of C I factors 
through EndAP0. This implies that the lnorphism ç is injective. Using the fact 
that Aa. is integrally close& we prove that ç maps Z. to A. Then our goal is to 
show that this mort)hism on the centers is an isolnorphism. 
In our original apl)roach ([12]), we showed that ç is an isonlorphism by proving 
sufficiently lnany geometric prol)ertics of ç to use the Main Theorem of Zariski. 
However, it was pointed out to us that a more direct approach might be to construct 
a morphisln 71 from Spec Za to b'pec A. using some of the ideas from [23]. Namely, 
we construct a natural tralsfornmtion betwcen fnnctors which arc represented by 
the schemes Spec Z: and .b'l)ec .4k. This then yields the required morphism. It is 
then easy to show that the composition r] o ç : Spec .4 -- ,Spec A,. is the identity 
morphisln. Using thc injcctivity of ç : Zk -- A., we sec that this is an isomorphism. 
Now we discuss the case when C(loes hot have a k-rational point. In this case, 
there exists a finite Galois extension k'/k such that Ck') is nolmmpty. The results 
discussed in the previous paragraphs show t hat after this base extension, the center 
Z, is isomorphic (via ç) to A.,, the coordinate ring of the conplement of the O- 
divisor in -Cc'/'  We show that this morphism descends to k. For any element 
er  Gal(k'/k), consider the automorphisn # = er - cç - ccrcç. Using a calculation 
of the universal bundle 7 ), we show that the restriction of its pull-back under p to a 
closed point y is isomorphic to 7). This then shows that it is the identitv morphism, 
giving us the required descent. 
The paper is organized in six sections. In the second section, we review some 
basic material about Jacobians of curves, keeping in mind the audience for this 
papcr. In Section 3. we study the restriction of the direct image of the universal 
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line bundle on the conlplenlent of the O-divisor. In Section 4. the inorphism ç is 
constructed and its properties are studied. In Section 5, we construct the morphism 
71. In the last section, the main theorem is proved, identifying the center of C I. 
In the following, we assume that the binar9 form f bas no repeated factors over 
an algebraic closure of k and that the characteristic of k does hot divide d. 

NOTATION AND CONVENTIONS 

 Ail rings have an identity elelnent. 
 Let B be an A-algebra and X ha a Sp«c A-schenle. V'e dellOt.e.¥ X Spec A 
Spec B by X XAB. 
 Ail schemes are locallv Noetherian and ail morphisms are of locally finite 
type. 
 (Sch/k) denotes the category of schemes over Spec k whose structure mor- 
phisms are locally of finira type. Also (Sch/X)fl denotes tire fiat site on X. 
Ail sheafifications are with respect to the fiat topology. 
 (fppf) = faithfldly-flat and of finite presentation. 
 The projections of fil»red products onto the i th comportent Ni are denoted 
by either pi,  or Px,, x,. 
 For any s«heme X and a positive integer n, .k () denotes the n-fold sym- 
metric prodn«t of X. 
 The t.erms line bmdles (respectively vector bundles) and invertible sheaves 
(respectively locally free sheaves) are used interchangeably. 
 For anv vector bundle E, (E) denotes the Euler chm-acteristic of E. 
 The translation morphism indu«ed by an element a of an abelian variety A 
is denoted by 
 All the fields m'e assumed to be infinite. 

2. THE UNIVERSAL PROPERTY OF THE PICARD SCHEME 
AND THE UNIVERSAL INVERTIBLE SHEAF 
In this section, we recall some baste properties of Picard schemes of curves and 
the universal invertible sheaf associated with them. The general references for this 
section are [2], Capter 8, and [15]. 
Let S be a base scheme, X an S-scheme and f : X -- S be t.he structure 
morphism. For any S-s«heme T, let q denote the projection X ×s :F -- T. The 
functor Picx/s which associates to anv S-s«heme T the Picard group Pic(X ×s T) 
is called the Picard fun«tor. This functor is hot representable, since it is hot a 
shee.f aven with respect to the Zariski topology. We consider its sheafification with 
respect to the (fppf)-topology (called the relative Picard functor). The sheafified 
functor is representable by a scheme which is also denoted bv Picx/s. If f has 
a section, the sheafified flmctor is isomorphic to the functor that assigns to any 
S-scheme T the group Pic(X ×s T)/q*Pic(T) (Proposition 4 in [2], Chapter 8). 
There is another description for the relative Picard functor for more restricted 
situations ([2], Chapter 8). We assume that f.(Ox) = Os holds universally (that 
is, this fornmla holds true after any base change) and that f admits a section 
e : S --, X. For a line bundle /2 on X, an isomorphisn c : Os -% e*(/:) is called 
a rigidification of/2. The pair (/2, cv) is referred to as a rigidified line bundle along 
the section e. There is an obvious notion of morphisns of rigidified line bundles. 
Then we consider the functor (P, e) : (Sch/S) ° --, (Sers) which associates to an 
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.5'-scheme T the set of isomorphism classes of line imudles on XT = .k XS T that are 
rigidified along lhe induçed section eT : T --+ X-v. The fimctor (P. e) is canonically 
isomorl)hic to the relative Picard flmctor Picx/s. 
Now consider a smoolh projective crve C' «)ver a field/," of genus g. We call the 
open (and closed) subflmctor that considers only invertible sheaves /2 on C x T 
whose restriction to C x {t} is of fixed degree n for anv point t the relative Picard 
fuctor of degree . Fhe schemc that represents it is denoted by Pic/. This is a 
smooth, l)rojective scheme over 
Next sui»pose thc curve C has a h-raIional point. Fix a section 
corresponding to such a rational point. Let (_9(1) be a vcry ample line bundle on 
Cof degree d. Then the s('helllePicc/ (I'eSl)ectivelv :d+g- 
 Pcc/ ) aise represents 
(p, e) (resl)ectively (pd+g-, ), which is an open subflmctor of (P, ) consisting of 
rigidified line imndles of dcgree (d + g- 1)). Se the identity on Picc/ (respectively 
"'c/k t givcs a line tmndle 7) (respectively P) on C x Picc/ (respectively 
C x 'lec/ ) that is canouically rigidified along the induced section. The sheaf 
Y is call«,d lhe tni,ersal (or Poinc(tré) line bundle for (C/1;, ê). In fact, it is easy te 
- d+9-1 
see lhat the correspondillg miversal line bundle on Ç x.. Æ'lcc/k is the pull-back 
of the universal line lmndle on C xt. Picc//¢ under the canonical inclusion. The next 
proposition, which is adapted from [2], Capter 8, 8.-1, Proposition 4, justifies this 
terminology. 
Proposition 2.1. With the notation as abot,e, the functor (P, e) is representable by 
a scberne which we denote by Picc/k. The miversal line bundle 7 ) bas the following 
property: For any k-schene X. and for ay line btndle F. o C× kX that is rigidified 
alon9 the induced section ex. there exists a miqte morphism 9 : X -- Picc/k such 
that F., as a rigidified line bmdle, i.s isomorphic te the pull-back of 7) under the 
morphism idc × 9- .4 similar staternent is truc for (pa+g-,e) if the line bundle 
£ Ic×{x} is of degree (d + 9 - 1) for any point x  X. 
Next we want te relate the schemes representing the relative Picard functors 
corresponding te C/k and Cl«/Iç. Here K is an arbitrary field extension of k and 
Ch = C ×te K. The next proposition is truc for arbitrary base extensions as well. 
See [7], Section 3. 
Let n : C -- Spec k be the structure morphism. Let Iç be any field extension of 
k and f : Spcc K -- Spec k be the corresponding base extension. We continue te 
use the saine notation as al»ove in the following proposition. 
Proposition 2.2. The Picard scheme Piccu//« is isomorphic te Picc/k ×k Ix" as 
a tç-scheme. Let 9 : Picc,//- -- Picc/k × Iç be such an isomorphism, and let 
7) be the universal line bundle for (C/k. e). Then hh(id × 9)*(P × id)*7) is the 
universal line bundle for (CI-/K. f o e), where the morphis-ms are described in the 
followin9 seque ce: 
h h 
C' x/,- Picc/K ----+ C x K x- Piccu//,-  C x K x Picc/. 
Reraarl 2.a. The first part of the above proposition remains true even if the curve 
C/k does not have a/,'-rational point.. Namely, the forlnation of the Picard scheme 
is compatible with the base change. This follows because, in fact, the usual con- 
struction of the Picard scheme is to construct it after a base extension so that the 
curve has a rational poillt (over the extcnded field), alld then to use descent. For 
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exainple, see [15]. Also, the above proposition remains true if we restrict to the 
open subflmctor of degree (d +.q - 1). This is clear since the degree of a line bundle 
does hot change under a pull-back via base extension of fields. 
Remark 2.4. We bave considered in the above discussion only the open subfunctors 
corresponding to degree (d +g- 1). But all the facts about representability relnain 
true for any degree. The facts about the universal line bnudle also remain true as 
long as the curve under consideration has a rational point over its base field. The 
proofs of these assertions are identical to the case of degree (d + g - 1). 
The Picard scheme is a group scheme (variety) over the be field. The Picard 
scheme of degree 0 is the Jacobian variety of Ç/k and is also denoted by J. The 
group operation corresponds to tensoring of line buudles. We define spific O- 
divisors in Picard schemes of degrees (g- 1) d (d + g- 1). 
Definition 2.5. Let C/k be a curve with the hypothesis  above (we do hot need 
C to have a k-ratiolml point). The O-divisor (of degree (g- 1)) is the schelnatic 
image of the canolfical lnorphism 
(C)-)  - 
 PICc/k, DT  [DT], 
where, for any k-scheme T and for any T-valued point Dç of (C) (-), [Dy] denotes 
the element of Pcc/ corresponding to 
Recall that the curve C is equipped with a (very ample) line bundle 0(1) of 
:d+g--1 
degree d. This gives a canonically defined O-divisor in Pc/ , as the image of 
the O-divisor under the morphisln 
 9- @(). d+9- 
PlCc/k  YmC/k . 
Now let K be a field extension of k. By considering the commutative diagram 

 g--1 
(CK)(g-1) --÷ PICcK/K 
(C)(9-1) > P1cc/k 

we see that the pull-back of the O-divisor under the morphisn # is the O-divisor 
for CK/K. In fact, the saine is true as well for the complelnent of the O-divisor. 
Proposition 2.6. Let C/k be a curve as belote, and let K be a field extension of k. 
Let  be a rigidified line bundle on CK of degree (d + g - 1). Then the image of the 
unique morphism Spec K J "-'" d÷g-1 
r'lCc/k corresponding to E lies in the complement 
of th« O-divisor if and only if h°(cK, E(-1)) = O. 
Proof. One direction is immediate once we note that the fibre over a closed point 
x under the composite morphism 
C(g-1 )  g--1 . d÷g-1 
-- PICc/k -- _t-'lCc/k 
either is empty or is exactly the linear system H°(C xk k(x), £x(-1)), where 
n- d÷9-1 
is the line buudle corresponding to the point x in rmc/ . This idea cau be 
formalized by using the Poincaré bmldle. 
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For the other direction, if b°(ch-,£(-1)) -fi 0, then w(, may choose a global 
section s of £(-1). Then the divisor Ds of this section gives an eleinent of C (9-1). 
Using thc Cmlonical morphism, this gives an invertible sheaf £2 on Ch- that has a 
global section whose divisor is Ds. Furthermore, the image of Spec K corresponding 
to/2 is bv definition in the -(0-divisor. But since their global sections bave the saine 
divisors, the invertit)le sheaves £(-1) and £2 must be isomorphic. This gives the 
other directions. [] 

We record an easy corollary for future reference. 
Corollary 2,7. l|ïth the hypothesis" as above, the pull-back (or the inverse image) 
. d+9--1 
of the complement of the (-)-dwisor in cc/k under the canonwal morphism 
. d+g-1 . d+g-1 
CcK/h  Cc/k 
. d+g-I 
is the complement of the (-)-divi.or in 
Remark 2.8. Note that our idca of thc proof of the corollary depends on C having 
a k-ratioml point. Howcver, by R('mark 2.3, the saine question can be ked if C 
(loes no have a k-rational point.  cosidcr a finite Galois extension k/k (with 
. d+g-1 
Galois group G) so that C(k ) is nonempty. Let C' = C xk k . Then PlCc,/k, h 
. d+g-I 
a G-action and gives, bv dcscent, rCc/ . The morphism 
(C,)(-) , . a+- 
 YICct/k 
is G-equivariant, and so the image of hk, is G-invariant. which descends to the - 
divisor in Pc/k . Hence the complemcnt of the O-divisor in YCc,/k, descends 
: _d+g- 1 
to the complement of the O-divisor in Pçc/k  
. d+g-1 
2.1. The universal line bundle on C k' mc,/k, - In this section, we would 
. d+g--1 
like to explicitly construct the universal line btmdle on Cxk, YCc,/k, . Though 
we consider only curves of degree d in 2, most of the calculations remain valid 
for arbitrary curves. Our goal is to find a formula which relates the universal line 
bundle to its pull-back mder a Galois aut.omorphisn of the base field. 
Ve start by recalling the construction of thc mfiversal line bundle for degree 0. 
Let C be a smooth curve over a field k such that C  = C x k £« has a k-rational 
point.  assume that k/k is a finite Galois extension. In out case, this follows 
from the assumption that the characteristic of k does hot divide d. To see this, note 
that thc curve given by (w d - f(u, v)) bas a rational point over the splitting field of 
the polynomial x d - f(a, b) for anv elements a, b  k. Let J denotc the Jacobian 
 0 
variety PCc,/k,. Fix a rational point P on C . Denote by 0 the theta divisor in J 
. 
obtained by translation of the usual -divisor in Pcc,/k, by ([-(g - 1)P]). This 
is the saine as the divisor of £()(-1)" £(-(g- 1)P), where O is the theta divisor 
o:d+g- 1 
in --'C'/k' dcfined earlier.  have the morphism ([15], Section 2) 
h:C' J, 
which maps a closed point Q to £([Q - PI). For any divisor D in t.he Jacobim 
variety, we denote by '(D) the line bundle m*£(D) OE p*£(D) - OE q*(D) - on 
J x k, J, where m is the multiplication morphism on J and p, q are the projections 
onto the first and second factors, respectively. Finally, write  for the image of 
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O0 under te morphism (-1)j  .1 -- ,1, and (-(00)a for 1«0 = 0 + a. Denote 
(o)« by 
With this notation, Lemnm 6.8 of [15] says the following: 
Lemma 2.9. The sheaf  = (h x (-1)«)*'(0) is isomorphic to the universal 
invertible sheaf given by the universal property of J. 
Using the line bundle of the last lelmna, we const.ruct the Ulfiversal line bmdle 
p: d+g-- 1 
for C' x, -c'/' " This construction is well known; see, for example, [1], Chpter 
4, Lelnln 2.2 nd thc discussion pl'eceding it. Denote by Lo the line bundle 
(-( - ))(-) 
O11 C t, alld bv 
is defin as 
p. +- 
lCc,/,  J 
L  L@ Lo. 
Ve can uow state the lemlna describing the univcrsa.1 bull(tlc of (lcgrcc (d + g - 1). 
Lemma 2.10. Witb the abot,e notation, tbe line bundle  = (idc, x q)*@p(L ) 
is the universal line bundle of degree (d + g- 1). 
Proof. Let  denote thc section of C   k  and let . «+«-, g denote the induced 
ICc/k 
sections. Then it follows that 
(idc, 
ICct /k 
This lneans that (idc, x q)*£ is rigidifiablc. Also, p(L ) is rigidifiable along 
:_a+-. Thus  is a rigidifiablc line bundle. Bv Remark 2.15, this is sucient 
--CC/k 
and we may nm.ke ally choice of rigidification. 
Now let T be aly k-schcmc alld let v be a rigidified line bundle on C  x, T 
such that for a.ny t  T, the restriction (T)t is of degree (d + g- 1). Then 
T = T @ Pi(L0 -1) is a rigidified line bundle on C  x k, T such that its restriction 
to any C; is of degree 0. This determilms a unique lnap 
and hence a map ç-I o   T  VlCc,/k, . Using the Ulliversal property of J a.nd 
 and the fact that the projection Pi COlnlnUtes with (idc, x ç), we see that the 
line bundle (idc, x (ç- o #))*ç is isomorphic to T. Furthermore. the morphisln 
ç-1 o # is unique with respect to this property. This is the universal property of 
Now let a be in Gal(k/k). If this induces an automorphism of a k-scheme, theu 
it is also denoted by a. For future purposes, we want to compute a*ç@ -. First 
we prove a computational lemma. 
Lemma 2.11. With tbe notation as belote, the following commutation relations 
hold: 
(1) 
(2) 
(3) 
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Pro@ Ve prove only the first relation, since the others are proved in a similar 
way. Let p.t (respectively p) denote the left. (respectively the right) side of the first 
equation. Let I = m o (#t, (-1) o #). Fix a closed point Q on C' and a divisor D 
- d+-I 
associated to a line imndle that corresponds to a closed point in rmc,/, . Let E 
denote the divisor of a section of £((g - 1)P)(1). Then 
pQ,D) 
= (Q.D) - (Q, D) 
= aOt(g_2)[p«-,_p ] o71o (h x (-1)j)(Q,D - E) 
- ,,,  (h  (-))  (i  ,)(Q, ) 
=   t(_)[«-,_  ,,,([Q - ç],- D) 
- ,,  (,  (-)«)(Q«, « - ) 
= «  t(_)[«-,_([Q - ] +  - 
= [Q« - w +  -  + (- 2)(ç- 

which is the cmpty divisor. (This follows from the observation that we may take 
E ° - E = (g - 1)[P ° - PI.) This means that p(Q, D) is the trivial line bundle. So, 
by the rigidity lemma ([14], Theorem 2.1), the two morphisms are the same. [] 

F):d÷g-1 
Now let 7 ) be the universal bundle on C' x, -":c'/' , and let ( be as above. 
r,. d+g-1 :d+g- 1 
Lemnla 2.12. Let x be a clo»ed point in rmc,/, and i  Spec k(x)  -'çc'/" be 
the corresponding morphism. Then. with the above notation, we bave that the line 
bundle (idc, x i)*(*ç@ç -) on C'x k(x) is trivial. 
. . d+g- 1 
is isomowhic fo pL, wheoE L is a line bundle oç ymc,/, and p is the projection 
onto the second factor. 
Pro@  first compute «*(idc, x )*£  (idc, x ç),£-x. To this end, consider 
*(idc, x 1)*£. Using the definition of £ and the relations of the lt lenma, we 
have t.tat 
* (idc, x 
= a*(idc, x )*(h x (-1)j)*£'((-)) 
: *(idc, x )*(h x (-1)j)*(m*£() @p*£(O) 
= (idc, x )*(h x (--1)j)*(m*£(OO)(g_2)[p«__p]+(g_l)[p«_l_p]) 
 p* (-(o0)«_,-P+( «)Ie«-'-Pl ) 
@ q* £(--(O0)(9_l)iP-, _pl+(g_l)IP.-'-pi )) 
= (idc, x )*(b x (-1)j) ( 
 £ - 
Now consider the line bundle m*((Oo)«_,_p]-O). By [14], Proposition 10.1, 
the line bundle ((O0)«_,_p]- O) lies in Pic°(Pic,/k,). But, by [14], Prop 
sition 9.2, for any line bundle ' on Pic,/a, that lies in Pic°(Pic,/,), we have 
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zn*£  -- p*£  ® q*£ . Combining this fact with the above calculation, we bave 

r*(idc, x 1)*£ ® (idc, x q).£-1 
= (idc, × ,)*(h x (-1)j)* 
*/2 - 
(p*/2((eo)eo_,_ 1 - (eo)-[o_,_l)®q 

-%-)). 

Recall that we are interested in the restriction of this line t)undle to C' xk, k(x). 
Since 

qo (h x (-1)j) o (idc, x l) o (idc, x i) = (-I)j OlOiO q. 

it follows that the pull-back of the q*-component of the above line bundle to C' xk, 
k(x) is trivial, and so we may consider only the p*-component. Note that 

po (h x (-1).) o (idc, x 1) o (idc, x i) = h Opl. 

This gives us 

(2.1) 

(idc, x i)*(cr*(idc, x ri)*/2 ® (idc, x 0)*£ -1) 

Now let h. (9) denote the morphisln (C') (9) -- Pic,/k,, which lnaps an effective 
divisor D of degree g to the line bundle associated to [D - gP]. Then it is clear 
that h(9)(P"-+(g - 1)P) = [p,,-l_p] and h(9)(gP "-') = g[P"--P]. So by [15], 
Lemma 6.8, the invertible sheaf h* £(((-)0)[p,_ _pl ) is isomorphic to £([P«- + (g- 
1)P]), and h*£((0)-[p«__pl )  £(g[P«-]). Using (2.1) and these remarks, we 
get 

(22) 

(idc, x i)*(cr*(idc, x ri)*£ ® (idc, x ri).£-l) 
- P*I£( P«- + (9 - 1)P - 9P«-I). 

Now we consider the line bundle (idc, x i)* (a'7)C.7)-1) on C'x , k(x). To finish the 
lemma, we consider t.he line bundle p(L0) -1. (Note that this pi is the projection 
r- d+g-1 
of C' x rcc,,/, onto the first factor. We contimle with this notation to avoid 
complicating our notation even further.) It follows from the definition of L0 that 

(idc, x i)*(cr*p(Lo) -1 ®p*(Lo)) 
(2.3) - (idc, x i)*£((g- 1)(P «-' - P)) 
 p;£((g - 1)(P «-' _ P)). 

Combilfing (2.2), (2.3) and the definition of P, we see that the line bundle 

(idc, x i)*(cr*P ®p-l) 

on C' x k, k(x) is trivial. 
The last part of the statement now follows froln the Seesaw Theol'em, Cor. 6, 
Sect. 5, Chap. 2 in [18]. [] 
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2.2. Auxiliary lemmas about rigidifications of line buudles. The following 
lemnm will i»e use(l ret)eatedly in lat.er sections. 

Lemma 2.13. Let C/k be a curve as before u,ih a k-rational point, e " Spec k --, C. 
Let Iç be a field ea'tcnsio of k. Then ay two line buTdles £-1, £-2 on Ch- that are 
rigidified along eh are isornorphic as rigidifled line bundles zf they are zsomorphic 
as line bundles. 

Another lelmna, which will be used later, coneerns line bundles on C xk D, 
whcre D = k[t]/(t ) is the ring of dual mtmbers. However, for later purposes, it is 
s/Icient to consider the case when k is algebraically closed. 
:_d+9--1 
Fix a section e  Sp«c k + C. " denote the induced section Pc/k  
:.d+ 9-1 n- d+ 9-1 
C xk P«c/k bv e._u+-. Let x be a «losed point of rcc/k , and let 
" C/k 

r. d+ 9-1 
Ix -Spot D ---+ 'mc/k 

be a tangent vector at x. This gives a rigidified line lmndle £1 on C xk D. Suppose 
that /2 2 is another line lmndlc on C xk D such that we have an isomorphisnl 

of line inmdles. Further suppose that £2 is isomorphic (as a line bundle) to j*/2 2 for 
some line bundle £-.2 on C, where j is the nmrphism Spcc D --, Spec k corresponding 
to the natural morl»hism/," '--, D. 

Lemma 2.14. Under the above hypothess and notation, there exists a rigidification 
on £-"2 such that £1 ad j*£-.2 are isomorphic as rigidified line bundles. Here J*£-2 
is rigidified usin9 the rigidification on £-'2 in a natural way. 

Remark 2.15. We omit proofs of these lenllllaS. The first lemma can be proved 
by conparing the set of rigidifications of £-2 with the automorphism group of £-2. 
The second lemma tan be proved bv comparing the sers of rigidifications of T', 
J*£-2 and £-2 with automorphism groups of (9 :^+- (gD and Ok. However, the 
proofs are easier if we use ver another definition of the relative Picard functor. 
From [2], Section 8.1, the relative Picmd functor is isomorphic to Ihe functor which 
associates to an S-schemc T the group Pic(XT)/Pic(T). In fact, this shows that 
two rigidifiable line bundles on XT that are isolnorphic are isolnorphic as rigidified 
line bundles. 

3. THE COMPLEIvlENT OF THE O-DIVISOR AND THE UNIVERSAL LINE BUNDLE 

In this section, we discuss the complement of the O-divisor. However, after 
some generalities, we consider only those curves which are of interest from the 
point of view of the Clifford algebra.. In particular, the projection of the re- 
striction of the universal bundle to C xk (complement of the O-divisor) on I? x 
(complement of the O-divisor) will be considered in detail. (The projection C --, I? 
is described below.) We continue with the notation and definitions from the last 
section. 
We first prove that the complement of the O-divisor is an affine scheme. This is 
an immediate consequence of well-known properties of the O-divisor. 
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Proposition 3.1. Let C/k be a degree d. genus g smooth curve in ]?2. (C need 
r,. d÷g-1 
hot bave a k-rational point.) Then tbe complement of the )-divisor in tqcc/k is 
an affine sch«m«. 

Pro@ First note that we moEv assume that C has a k-rational point. This is because 
r. d+g-- 1 
the O-divisor and its conlplement were obtained by Galois descent on cc,/k, , 
where k'/k is an extension so that C(k') is nonempty. 
r. d+g-- 1 
Suppose Chas a k-rational point. Then the O-divisor is ample in lCc/ . 
Sec [15], Renlark 6.5 and Theorcm 6.6. Now the proI)osition follows, since the 
.. d+g- l. 
conlplelnent of any ample divisor in VlCc/k ls an affine schelne. [] 

Remark 3.2. If C has a k-rational point, then the saine proof shows that the conl- 
plement of any -divisor in Pic/k is aflïne for anv m. 

Notation. Foi" any curve C/k as a.bove, wc (tcnote by Spec Aa the open complemcnt 
r. d+9--1 
of the (-)-divisor iii VlCc/k . 

For the rest of this section we assume that C bas a k-rational point. 
Recall from the introduction that we are mainly intercsted in the curve C = 
Proj(k[u, v, w]/(w d - f(u, v)). (Hcre f is the given binary fornl.) We prove that in 
the cases of interest, the curvc C is always nonsingular. 

Lemma 3.3. The curve C is nonsingvlar provided that f does hot bave repeated 
factors over an algebraic closure of k and that the characteristic of k does not divide 
d. 

Pro@ We nloEV assulne that k is algebraically close& An easy conlputatioll shows 
that 
0(' e - f(,v)) Of O(w e- f(,v)) _ Of 
Ou = - -u ' Or 0 v 
and that °('e-I(")) - dw d-. Also we bave that u + v = d9 for anv poh'- 
nomial 9(u, v). Since the characteristic of k does hot divide d, C is singular only 
if the system of equations {f, ol ol 
 0' o } has a hltion. However, since f has no 
repeated factors, this is impossible.  

Corresponding to the in«lusion k[u,v]  (k[u, v,w]/(w d- f(u,v)) we have the 
projection 
For any field extension K/k, we denote the morphism obtained  above by qu- Let 
y be a closed point in Spec A, and let k(y) be its residue fiel& Then we denote 
by £ v the pull-back of the universal line bundle  under the canonical norphisln 
. d+g- 1 
C x k(y)  C x U1CC/k  
The following proposition is a key proposition for our purposes and was proved in 
[22] when k is algebraically closed. We give a proof for the sake of completeness. 
Proposition 3.4. The coherent sheaf  = (qk(y)).£y is isomorphic fo the trivial 
vector bundle d ()" 
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Pro@ We first assume that k is algebraically closed. Since lP)¢ is a nonsingular 
curve over an algehraically closed field, any torsion-free coherent sheaf is a vector 
bundle. Since (q). respects torsion-freeness ([6] l. Proposition 8.4.1),  is a vector 
bundle on P. Since any vector bundle on P is a sure of line bundles, we can write 
d , 
  = Oef (ui) for some integer 'i s. Note that k(£) = X(). Then, by [5], 
Cor. 15.2.1, 
k() = (rank(u))(1 - 9e) + deg(), 
Here deg(g) denotes the degree of a vector tmndle . This gives i ni = 0. Since 
g is in the complcment (ff thc (-)-divisor, we have 
h°(F,(q).(£y@oc Oc(-1))) = h°(F,(qk).(£y(-1))) 
= °(c,(-)), 
which is zero. But by the projccti(m formula, we have the idcntity 
(q»).(uNOc (q)Oe(-1))  (q).u No Oe(-1). 
Using this and the prcvious equations gives 
i 
so ui  0 for all i = 1,-.- , d, and hence ni = 0 (since , u = 0). 
So (qa).£u  a 0. Now let k be any fiel& and let y be a point  before. 
Then we consider the (canonically rigidified) pull-back of the line bundle £u under 
the canonical morphism C xa   C x k(y). Then by Corollarv 2.7, the image 
of the point corresponding to j£u lies in Spec Ai.. By the earlier part of the 
argument, (q).(j(£u)) =  Oe. To finish the proof we consider the following 
commutative diagram: 

(3.1) 

Now (q),(jS(£u)) - (j,)(q(u)).£u by [lll, Chapter III, Theorem 9.3. But 
Aut((3,,)(q(u)).£u) = GLd(f¢) and Ht(Gal./t¢(u), GLd()) = 0 by Hilbert 90. So 
it follows that .T u - (d O(). [] 

Next we want to consider the direct inmge of the universal line bundle on C ×te 
;d÷g- 1 
Pc/k mader the projection onto the second factor. In general, there ib no reason 
for this to be a vector bundle. For our purposes, the restriction of this direct image 
to Spec Ae is more important. With the help of the last proposition we prove the 
following. 
r,. d-i-g- I r. d÷g- I 
Proposition 3.5. Let 7r  C xa llCc/ --, r'cc/ denote the projection, onto 
the second factor. Then (7r,7 9) [sp«« A» is a locally free .sheaf of tank d. 
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. d+g-1 
Proof. First note that since Spec Ak -- rlCc/ is an open imlnersion, 
(7 Ic×spc A)  (.ç) sç « . 
For any point y in Spec A, let k(y) be the residue field of y. We denote by Py the 
. d+g-1 
line bundle (id x i)*P on Ck(y), whcre i is tlle morphism Spec k(y)  WCc/ . 
We will prove that dimk(y)H°(Ck[), Py) is constant. Since Spec Ak is an open set 
:d+g-1 
in an integral scheme Pc/a , it is itself irreducible and reduced. Then by [18], 
Corollary 2, p. 50, it follows that (u,) Isc A is a locally free sheaf of rank d. 
First we considcr a closoE point y in Spec A. Theu, with the notation from 
diagram (3.1), (qa()). is a trivial tank d vector bundle. So 
hO(c(), p) o 
=  ((), (q()),P) = a. 
This holds for all closed points in Spec Aa. By the upper semicontimfity of 
the function y  dilnk(y)H°(Ck(y),y) and the fact that Spec A is a Jacob- 
son scheme (in particular, any open set contains a closed point), it follows that 
dim() H ° (Ca(), P) is constant. 
The locally free sheaf on Spec A in thc statcment of thc previous proposition 
will be denoted by £. As part of the proof of the lt proposition we provcd that the 
function y  diln()H ° (Ca(), P) is constant. This, Colnbined with [18], Corollary 
2, p. 50, gives the following corollary. 
Corollary 3.6. For all y  @ec A, the natural map 
is an isomorphism. 
We have the following sequence of morphisnls: 
qA k PA k 
C x  Spec A   « Spec A  Spec A. 
We continue to denote by u the projection onto the second factor of C  Spec Aa 
(so that PA cqA« = ). Let  denote the cohcrent sheaf (qA), on  Xk Spec Aa. 
The following lemma justifies this notation (compare Proposition 3.4). 
Lemma 3.7. For any clos point y in Spec Ak, let i : Spec k(y)  Spec Aa be 
the corresponding inclusion. Consider the Cartesian square 
idxi 
C x Sp«c k(y)  C x Spec A 
qk() qAk 
Then the coherent sheaves (id x i)*(q),P and (q()).(id x i)*P are isomorphw. 
Proof. Since P is a coherent sheaf on the projective scheme C x  @ec Aa (over 
@ec A), there exists a graded (A[u, v, w]/(w  - f))-module M such that M is 
isomorphic ( an Ocx«sp A-module) to P. Then the sheaf (id x i)*(q()),P 
is isomorphic to (()[,](k(y)  AM)) . Also the sheaf (q()),(id x i)*P is iso- 
morphic to the sheaf (k(y)  A[,]M) . These conclusions follow ri'oto [6] 
Proposition 2.8.10. But the graded k(y)[u,v]-modules (()[,](k(y)A AM)) 
and (k(y) A A[,]hl) are isomorphic. So the sheaves in question are isomor- 
phic. 
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Our goal for tllis section is to find a graded Ah[u,v]-module whose associated 
slleaf is isomorl)hic to . In fact, we will compute the module associated to , 
which will be sufficient. The next proposit.ion is the main tool in this colnputation. 
Recall that for any nlOrl)hisnl g : X --* }" and a sheaf Ç of Ox-lnodules. there is a 
natural morphism 9*g.Ç --* Ç- This follows from the adjointness of the filnctors g* 
and g.. 

Proposition 3.8. Vïth tbe notation as before, tbe natural morphism 

u " P*A (PA»).."  ." 

is an isomorpbism. 

Proof. th,call that (pA).f" ----- £ is a locally fl'ee sheaf of rank d. So the sheaf 
P. (P,4),U is a.lso a locally free heaf of rank d on 1?à (1?à = 1?. x a Spec .4). 
First we clailn that ,T is a locally free heaf of tank d on 17. We prove that 
dilnk(u)f" ®oà k(y) is d tbr any closed point y in 1?à. [ndeed bv the upper 
SelniColltimlity of the dimension flnlction and bv the fact that 1?!4 is of finite type 
over k, this will |)e sufficieut. 
Now for any closed point y in I? 1 consider the following diagram: 

Pk() IPAk 
Spec k(y)  Spec Ak 

/3y the universal property of the fibre product, the dotted arrow i exists so that 
this is a commutative diagram. We prove that 

is d. But 

dinlk(y)ff ®oà. k(y) = dima(yj*ff 

dilllk(y ) j*9 c = dilnk(u) i*(id x lu)*9 c. 
Prom Lemma 3.7 and Proposition 3.4, it follows that (id x i)'9 c is a trivial vector 
bundle of tank d, so that dinlk(y) i*(id x i)'9 c is d. 
Now we go back to the morphism u defined earlier. Notice that it is a lnorphism of 
vector bundles of tank d. To prove that u is an isomorphism, it is sufficient t.o show 
that u (PA (PA).) ----'  is bijective for all points x in 1?À.. Bv [6] I, Corol- 
lary 0.5.5.7, it is sufficient to prove that u is surjective at closed points. The pas't 
about closed points follows fi'om the fact that the set {x e 1?À I u is surjective} 
is open in 1?À. Then by [6] I. Corollas'y 0.5.5.6, it is sufficient to prove that 
u u ® id: (P*A (PA»)*f')v/mV(P*A. (PA)*f')U  f'u/muf'u 
is surjective for all closed points y in 17!4- But this homomorphism is surjective if 
and only if the morphism 

j*u " j*P.*4 (PA).f"  j'f" 
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is surject.ive, which is the saine as 
i*(id x iy)*u" i*(id x iy)*p*A(p.4»),. --- i*(id x i)'9 r 
being surjective. Soit is sucient to prove that 
" *  i x'p* (idx 
(idx) u (idx u A»(PA)* " * 
is an isomorphism. But we have the isomorphism 
(id x i)*p.,.(pA),Y  * i*" 
= P() 
Note that y(P.4.), is a trivial vector bmdle of rank d, and so 
d d 
This gives that (id x iu)*p  (P.4), is a trivial vector Imndle. Hence the morphism 
(idx iv)* will be an isolnorphism if it is so on the global sections. But the morphism 
on the global sections is an isomorphism if the natural morphism 
H°g 
is an isomorphism. By the earlier part (dimension computation) of the proof 
and [18], Corollary 2. p. 50. this is indeed the case. 
A useful application of this proposition is that it gives a convenient way to de- 
scribe the graded module associated to the sheaf . Consider the sheaf (p&.),U = 
u,(P) on Spec A. As we saw before, this is a locally ri'ce sheaf of rank d on 
Spec A.. So we tan find a projective A-modnle P such that 
Corollary a.9. The A[u,-]-oat« O, (e.«,,*(0)'  (ve«e) isomorphic to 
the module M = P Na A[u. ri. 
Pro@ It. is sufficient to compute the graded module sociated to the coherent 
sheafp* (PA»),, which we denote bx ç. But. bv the projection formula, 
(PA),(Ç(i))  (P.4),(U)  (»A),(Oà(i)). 
Since 4  Spec A. is a projective morphism, the module in the statement, tan 
be obtained by computing H°(Spec A, (pA),(Y) (pA),(O(i))) for any i. 
From the above isomorphism it follows that 
i 
But now the statement follows from the well-known computation of the graded 
module sociated to the structure sheaf of the projective -space. Sec, for exam- 
pie, [11], Chapter V, Proposition 5.13. 
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4. THE HOMOMORPHISM ("ïl  End,4(P) 
We recall the set-up of the reduced Clifford algebra. Let f be a binary form 
of degree d over a ficM k such that char(k) does hot divide d. Then the Clifford 
algebra of the form f was dcfined in the introduction. Sec Section 1. One of the key 
properties of the representations of C Iis that the dimension of the representation 
is divisible by d ([10], Proposition 1.1). We are interested in representations of C 
,ff dimension d. We form the redu«ed Clifford algebra C I by (Ci)/(Çp) , where the 
intersection is takcu over the keruels of ail dimension d rcpresentations. The algebra 
C I is in fa«t an Azumaya algcbra (for example, [10], Proposition 1.4), and so the 
d-dimensional represeutations are paralnctrized by the prime ideals in the tenter 
of C I. Furthermore, it was poiuted out to us that the center of C I is Noetherian. 
This, for example, follows from Proposition 2 of [17], since CI is finitely generated 
over k and C I is a finite module over its tenter. We are trying to prove that the 
conter of C I is isomorphic to the k-algebra A of Section 3. 
In this section, we consider lhe case when the curve C of Section 3 has a k- 
rational point. Wc construct a homomorphism from the tenter of C I to Ac. This 
is achieved by first constructing a homomorphisn from C I to EndA(P), where P 
is the module of thc global sections of the sheaf r.7  ou Spec A. Sec Section 3. 
We will show in the following sections that under this holnonorphism the tenter 
of C I is mapped to 4. After we have obtained the desired homomorphism, we 
iuvestigate some of its properties. In particular, we want to show how, starting 
with a finite-dimensional representation of C I, we tan construct a line bundle over 
C (possibly after a base extension). 
Before we get to the main proposition of this section, we note an algebraic 
relation which holds inside the (graded) endomorphism ring of the graded module 
in Corollary 3.9. This relation will then enable us to define the morphism mentioned 
in the previous paragraph. 
For the test of this section, we assume that the curve Cbas a k-rational point. 
Lemma 4.1. Let P be the projective A-module as in Corollary 3.9. Consider the 
grad«d Ak[u, v]-module 1I = P ®A A[u, v]. Let P denote the ith graded piece of 
this module (so P - Po). Then we bave the following equation: 
HomA (P0, Pi) = u o EndAP0 + v o EndAP0, 
where u. v are viewed zn HOmA(P0. P) with their natural action. 
Pro@ The proof is easy and follows from the relation PI  (Po  Po), where the 
two generators are u and v. [] 
Proposition 4.2. There exists an algebra homomorphism 
ç :C I -- End.4 P, 
where C I is the reduced Clifford algebra and P is as defined belote 
Pro@ The strategy to prove this proposition will be to show that EndAP has 
elements that satisfy the relations of the Clifford algebra. 
Consider the universal line bundle 7  on Cx A and the projection qA : C xk 
A - ItÀ. Since qA, is a projective (and hence an affine) morphismç we tan obtain 
the graded module associated to 7 
H (A,(qA).(7(i))). But 
this was already computed in Corollary 3.9. So the module M of Lemma 4 1 is 
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also a graded A.[u, v, v]/(a, d - f)-module. Since t« is a honlogeneous element of 
degree one in this (naturally) graded ring, it is an element of Hom.4» (P0, P). Bv 
Lemma 4.1, there exist elements a, a¢. in EndAP such that the equality 
holds in HomA (P0, P). Now consider the element w d as an element of the module 
Hom.4(P0, Pa). Since in the graded ring of C xe Ae we have the relation w a = f, 
it holds in Homa (Po. Pd) as well. In particnlar, we get 
u'd = (u o a, + v o o,,) a = f(u. v)Id. 
But note that Holn.a(Po, Pd) is a froc End.4.P0-module. This shows that the 
elements o. and a,, in EIIdA(P0) satisfy the relations of the Clifford algebra CI 
and give a homomorphism 
ç : CI  EndA,.P0. 
Next we want to show that this homomorphisn factors through the reduced Clifford 
. d+g-1 
algebra. Recall that Sp«c .4e is an open subschelne in rlCc/k , which is integral. 
Thus Spec A is an integral scheme of filfite type over k. In particular, A is an 
integral domain. So we may consider the sequence of morphisms 
Ci  Enda»(P0) EndQ(A)(P0 @A Q(Ak)) ld(Q(Aa)), 
where Q(A) is the field of fractions of A. Note that, by the comments ruade 
above, the second arrow is an injection. This gives a d-dimensional representation 
of Cf. So the composite factors through Çy.  consider the diagrmn 
Cf  End.4 P0  Md(Q(Ae)) 
The dashed arrow exists since the second horizontal arrow is injective. This gives 
us the desired holnomorphism 
ç : Çy  EndA P0- 
The homomorphism Çy  End.4» Po constructed in the last proposition will al- 
ways be denoted by 
Next we want to stndy the homonorphism ç. To this end. we assume that we 
are given a representation of 
where A" is a field extension of k. Our goal is to show that any such representation 
factors through Ends4» P0 via 
We denote the images of the two generators of Çy under  by ff and ff,,. Consider 
the morphism 
(*- f) 
,   [], [l, 
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Lennna 4.3. fit is an im,ertible shea] on X. Moreover, the pull-back of this sheaf 
to Ca under the canonical morphism is a d«gree (d + 9 - 1) line bundle on CK. 
. d+g- 1 
The image of the uniqae morphism Sp« c K -- t'lCc/ correspondin9 fo this line 
bundle lies in Sp«c .4. 

Pro@ For the first part of thc statemcnt, by [6], 4, Proposition 2.5.1. it is suflïcient 
to prove lhat i" is invertible with the assmnption that K is algebraically closed. 
We will prove that for any closed point .c G X, dimh (" ®O-. K) = 1. This 
will i)e suflicient by the usual upper semicontinuitv argmnent. Furthennore, it is 
clcar that boh u and v calmot be in a homogeneous maximal ideal of Sh. So 
it will be suflïcient t.o tre)ve the dim(,nsion condition for any closed point x in 
X, = Spec((S«)()), since le argument for .\', is similar. Now 

(S,-)(,)  (ff#_ f(5, 1)) 

d 

where ,2 = a/v and iV = w/v. Here  acts in a natural way and  acts as 
Any closed point in Spec((S,,-)(v)) is m = (i - a,@ - b) for some a, b /x. So we 
have that 

(gSPec((SK)(')) "x - (ff, d _ f(ii, 1)) (a-c. -b) 

as well as 

d 

This gives 

(). ®o. ç  @e Iç[] 
(f - a, ï, - b)(@aK[f]) 
(ocf + tï, - b)(@dI()" 

So the required dilnension is dimK(ker(aff,, + o. - b)). We consider aoT,, + e2,  
Ma(K) and compute the dimension of its eigenspace of eigenvalue b. The character- 
istic polynomial of aoT,+o-, is ta-f(a, 1). Indeed, this follows when f(a, 1) = 0 since 
all the roots are distinct (since char(K) does not divide d), and when f(a, 1) = 0 
since the matrix is nilpotent in this case. Moreover, if b =/= 0, then f(a, 1) =/= 0 and 
b is an eigenvalue of lnultiplicity 1. 
Let b=0, sothat f(a, 1) =0. We mavfind amatrix B  GLa(Iç) such that 
B(aoT + cï,)B -1 is in it.s Jordan form. If dimK(ker(aa-,, + e2. - b)) > 1, then we 
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can write (for w e lId(K[u, v])) det w = det BwB -1 = (av - u)ldet w', where 
[av- u ... 0 0 ... 0  

BwB- 1 

0 ... av-u 0 ... 
0 ... 0 1 ... 
0 ... 0 0 ... 

There are l entries (av - u) in the above matrix, and l _> 2. But. this gives that 
det w d = (av - u)lddet(w') d = f(u, v) d. Since l _> 2, aï,- u is a repeated factor of 
f(u, v), which contradicts our assumption on f. This gives the required dimension 
condition, and/ is an invertible sheaf on CK. 
For the second part, consider the projection qK " CK -- PK considered before. 
Then 
\(c,-, )= (},, (q,,-).), 
and, by the Rielnann-Roch formula for vector bundles on nousingular curves ([5], 
Example 15.2.1), 
(1 - 9) + deg/ = d(1 - 0) + deg ((qK)./). 
But (qK)* - (aO} and so deg ((qu).) = 0. This gives that deg  = 
(d+g- 1). 
For the third part, note that by the projection formula, 
h°(cu, (-1)) = h°(P}, ((qu).)(-1)) = 0. 
By Proposition 2.6, the image of Spec L" under the unique morphism corresponding 
to N lies in Spec A k. [] 

Remark 4.4. The above lemma is essentially the key lemma of [22]. As we will see 
later, it provides an exp|icit construction of line bundles on C ×k K arising from 
representations of Cf. It is the main tool in proving the one-to-one correspondence 
between representations of Cf and line bundles on C mentioned in the introduction 
(Section 1). 

Now we get to the nmin proposition mentioned at the beginning of the section. 
Proposition 4.5. Let   Cf -- Md(K) be a representation. Then there exists a 
morphzsm EndAP0  lId( Iç) so that the following diagram is commutative: 

> EndA P0 

Proof. Given a finite-dimensional representation of C I, we can construct a graded 
&-odule N as b«ore, where & = (h'[, , ]/("- f)). This gies a ine bundle 
£ on Cu as in the last lemma. Then the image of t.he morphism corresponding to 
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this line bumlle lies in Spec At. Note that by Lelmna 2.13, this morphism does hot 
depend on the «hoiçe of rigidification. If we &,note by i the morphism 
Spec h  Spic A, 
then (id x i)*  £, where  is the mfiversal line bundle on C xk Spec Ak. 
Recall that 
tt°(C Xk Spcc Ah., P(i)). In fact, 
So the liue lmndle (id x )*  £  ', where " is the graded SK-module 
K @A (, Pi)- k claire that the Sh--modules N  and N are isomorphic as graded 
modules. 
br any line lmmlle £ , we dem»te by F«(£ ) the asociated graded module 
, tI°(£'(i)). We have lhe [ollowing diagram, in which the horizontal arrow exists 
by fimctoriality of V. and the vertical morphisms are natural: 
Since the slmaes associated to the graded modules N. N' are isomorphie, the 
horizontal arrow is an isomorphism (as ç*(OCK) and hence as S-modules). Thus 
it will be sueient to prove that 
(1) X  F.(N') and 
(2) s' 
as SK-modules. Note that the vertical morphisms are S-module maps, and hence 
it is sucient to prove that these are bijective. Consider the first morphism as a 
film ri-module morplfism. By [11], Proposition 5.13 and the fa«t that F. comnmt 
with formation of direct. 8tilll, it follows that the first morphism is an isomorphisnl. 

But a sinlilar argmnent along with the observation that 
N' - (K ®A Po) ®A, Ah-lu. v] 
d 
d 
shows that N'  F.(") is bijective. 
This gives a degree-preserving SK-module isomorphism 
d 
In particular, we get a K-module isomorphism 
d 
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and hence an isomorphism 
lId(Ix') uZ End,-(/x ®Ak P0)- 
Since/t preserves the action of w, the image of OE (respectively Cfv) is (id ® a) 
(respectively (id ® av)). So we have a «omnmtative diagram 

C I  Endos. Po 
IIld(tX-)  EndK(K ®A Po) 

This is the desired diagram of the statenlent. 

Remark 4.6. Note that the morphisln EndA,. Po -- Md(I() obtaincd in the previous 
proposition by our procedure is uniquely determined up to a Ulfique automorphisln 
of Md(I(). In fact, the Olfly place where we do hot have an explicit construction is in 
the use of the morphism Spcc K -- Spec Ak. However, this morphisln is uniquely 
determined once we have constructed a line bundle N as in the proof. This follows 
by Lelnnm 2.13. The rest of the procedure is algorithnfic, and deterlnines a unique 
honlomorphisn up to an element of GLd(/(). 

We now use the above proposition to prove a very useflfl property of the mor- 
phisn ç. 

Corollary 4.7. The morphism 
ç: C I  EndAPo 
constructed earlier is injective. 

Pro@ Let I denote the kernel of the lnorphism ç. Then since any representation 
has a factorization as in Proposition 4.5, I must be contained in the intersection 
of the kernels of d-dimensional representations. Thus, since C I is an Azunmya 
algebra, I is contained in the (two-sided) idem generated by the nilradical of Z, 
the center of C I. But the center Zk of C I is reduced; sec [10], Proposition 1.4. So 
the ideal I = (0), and ç is injective. [] 
4.1. The morphism ç : Spec Ak  Spec Zk. Consider the lnorphisln ç : C I -- 
End&.P0 as above. 
Proposition 4.8. The morphism ç : C I  EndAP0 maps the center Z fo A. 

Pro@ Consider the sequence of morphislns 
- i 
C I  EndAk Po " Md(Q(Ak)). 
Let z  Zk. We claire that ç(z) is integral over Ak and that i o ç(z)  Q(Ak). 
The first claire follows since ç(z) commutes with Ak and EndAP0 is a finitely 
generated module over Ak. For the second claim, note that the composite i o ç 
is an irreducible representation of C I. Hence it follows that (i o ç)(CI)Q(A) = 
Md(Q(Ak)). Now the claire follows since this implies that i o ç(Zk) _C Q(Ak). 
But since Spec Ak is a smooth integral scheme, Ak is integrally closed, which 
implies that ç(z) ¢ Ak. [] 
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The ai)ove proposition aiong with Corollary 4.7 gives an injective k-algebra mor- 
l)hisn 

and hence a k-morphism 

o:Zk ----+ Ai. 

ç : Spec Ak --- Spec 
ff there zs no cause for confusion, we will denote both of the above raorphisras by 
'We will prove in thc following sections that ç is an isomorphism. 

5. THE MORPHISM q :Spcc Ak --+ Spec Z 

In this section, we construct a morphism q : Spec Z -- Spcc A which we will 
show te bc the inverse of thc morphism ç constructed earlicr. The main idea in 
constructing the morphism q is te vicw thc s«hcmes Spec .4k and Spcc Z as rep- 
rcsenting objects of certain flmctors, and then contruct a natural transformation 
betwccn these functors. Thesc idcas are prcsent in the Imper [23] of M. Van den 
Bergh. 
If Sis a k-scheme, then i)y an S-representation of degree n of C I we mean a 
pair (b, COA), where COA is a sheaf of Azumaya algebras of tank n 2 over S and lb : 
C I --) H°(S. (gA) is a k-algebra homomorphisnL Two S-represent.ations (b. 
and (Ib2, COA=) are said te be equivalent if there is a.n isomorphism 19 : 
of sheaves of Azumaya algebras such that Ib2 = H°(S, 19) o b. An S-representation 
of ÇI is irreducible if the image of ÇI generates (9 A locally. 
Let 7ePd(Çl,-) be the functor that assigns te a k-scheme S the set of equiv- 
alence classes of irreducible S-representations of degree d of ÇI- Since AzumoEva 
algebras pull back te Azumaya algebras and irreducible representations are stable 
under pull-back, it follows that this is a functor. Furthermore, since ÇI is an Azu- 
maya algebra of rank d 2 over its center, this functor is representable in Sch/k and 
is represented by Spec Z¢. Note that, bv remarks at the beginning of Section 
Spec Z,¢ is a Noetherian scheme. 
We also consider representations of C7 I into endomorphism sheaves of vector 
bundles. Let Ça(C I, -) be the subfunctor of I'epd(C I, -) that assigns to a k-scheme 
S the set of equivalence classes of irreducible S-representations into endomorphism 
sheaves of vector bmdles of rank d. Again, siuce endomorphism sheaves of vector 
bundles pull back to the sheaves of the saine kind, it follows that this is also 
a fun«tor. Both the fun«tors 7epd(Ci,- ) and Çd(CI,- ) define presheaves on 
(Sch/k) with respect te the fiat topology. Moreover, the functor Rep,(C I, -) is 
a sheaf with respect te the fiat topology, since it is representable (Proposition 1, 
Chapter 8, [2]). We denote the sheafification of the functor ç(O-) with respect 
te the fiat topology by ç(CI,- ). The following fact is [2ai, Lemma 4.2. 

Lemma 5.1. The natural transformation Ça(Ci,- ) --) Tepa(C,-) induces an 
isomorphism gep((Ç I, -)  ç,(Çl,-)- 

Our next goal is te define a natural transformation frein 7epa(Cy , -) te a certain 
. d+g- 
open subfunctor of oeCcl  (-). Before we define this morphism of functors, we 
need seine auxilia.ry lemmas. 
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Let S be a k-scheme, and let (b, OA) be ai1 element of Çn(CI, S) so that (-9A = 
,fndos(,f) for solne vector bundle g of rank d on C x S. We will construct an 
invertible sheaf on C x S. Consider the graded sheaf homonorphism 
Os[u.,w] 
, _ ./'(u, ,) 

,,--- u¢(xl) + ve(x2), 
where xl and x2 are the usual gencrators of ÇI- Since  is a sheaf of s-modules, 
there is a canonica] map from ». to d(). Since this is a gradcd homomorphism 
of gradcd algcbras (degree of u. v is 1 on the right side), we get a graded module 
 @os s[u. v]. This defines a shcaf  on C k S. In the lemma bclow and in thc 
remainder of this section, for any point, s in S, q dcnotes the morphism Ckk(s)  
F(s) induced by the inclusion k(s)[u, v]  (k(s)[u, v, lu])/(, d- f(u. V)), p, denotes 
the projection of the second factor of  onto Spec k(s), and s denotes the 
(s) 
composition Ps  q. Similar uotatiou is used for an arbitrary field lç instead of 
(s). 
Lenlma 5.2. The sbeaf  is an int,ertible sheaf on C x S of (fibre-wise constatt) 
de9ree (d + 9 - 1). Furthermore, the invertible sheaf  =  @os Spec k(s) on 
c¢) stisfies h°( C¢),  (-  ) ) = O. 
Proof. It is sufficient to prove the first assertion in the case when S is an affine 
scheme. So let S = Spec R. Let C denote the scheme C x S. XX nmy assume 
that H°(S. E)  d R, since E is a vector bundle of rank d. In this case, we denote 
the graded (R[u,v,w]/(w d- f@.,v))-module H°(S,E) @ R[u,v] by M and the 
sociated sheaf  by M. 
Now consider the first assertion. Note that I is fiat over S and  is a fiat 
morphism. So by [6], 42, Lemma 12.3.1, it is sufficient to prove that for anv point 
s in S,  is an invertible sheaf on S. Here I is the sheaf (1 x i)*l where 
1 x i is the morphism C = C x Spec k(s)  Cx S. Consider the representation 
sociated to the point s via ¢: 
" d  M,(()). 
This gives a sheaf  as in the discussion preceding Lemma 4.3. The sheaf  is 
isomorphic to the sheaf I. So the fact that M is invertible now follows from 
Lemma 4.3. Also, the degree is (d + 9 - 1) along fibres of the projection onto the 
second factor. This follows from a calculation using the Rienlann-Roch formula« 
see the proof of Lemma d.3. 
For the second assertion, it is sufficient to prove the statement for the invertible 
sheaf  of the last paragraph. But, in this ce, (q()). is a trivial vector bundle 
of rank d. This, along with the projection formula, gives the required statement. 
See Lemma 4.3 for details.  
Crollary 5.3. With the notation as in Lemma 5., the vesulting morphism S  
P'C/k:d+g- factors through Spec Ak. 
. d+ 9-1 
Proof. Since the sheaf  is invertible on CS, we have a morphism S  VlCc/k 
(Mter rigidiçving this invertible sheaf if necessary). om Remark 2.15, this mor- 
phism does hot depend on the rigidification. Now it is sufficient to prove that the 
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inmge of anv point s in Sis in Spcc Ak under the morphism in the statement. 
However, it follows ri'oto Lemnm 4.3 tlmt the image of any point s in S lies in 
Spe« Ak. So we get the required fact.orizatiolL [] 

 7_[d+g- 1 
Now we consider mlother functor, Lc/ detined as follows. This will be defined 
n- al+g--1 Ne+g-l/s) be the subset of 
as a subfunctor of rlCc/ . For any k-scheme S, let "'c/k 
p. d÷g--1 
Cc/ (S) associated to the set of rigidified line bundles/2 on C xk S that satisfy 
thc conditi(m that for any point s of S, h°(Cs,/2s(-1)) = 0. Here Cs = C xk k(s) 
and /2s (lelotes the sheaf (1c x i(s))*£, where i(s) is the inclusion Spec k(s) --, S. 
To sec that this is ind(,ed a flm('tor, consider a k-schelne T and a k-morphism 
f : T --, S, and let £ i)e an invertii)le shcaf on Cs = Cx S which gives an element 
 d-t-g- 
of Hc/k (S). We need to prove that for any t)oint t in T, h°(Ct,£t(-1)) = O. 
But if s is the image of t, then we have h°(C,,/2(-1)) = 0. Now the required 
dilnension is zero, since the COlnposition f o i(t) factors through the inclusion i(s) : 
Spcc k(s)  S. It follows that this definition gives a flmctor. For this flmctor, we 
havc the following lennlla. 
Lemma 5.4. The functor  d+- 
l-lc/is an ope subfunctor of the functor '" d+g- 
rCc/  
 d+g-1 
Furthermore, l-teW is a represetable functor and zs represented by Spec 
q_ld÷ g--1 
Pro@ To verify the openness of "c/k , we use the characterization of openness 
of subflmctors given in [23], Lenulm 2.2.3. For a k-scheme S, let/2 be an invertible 
sheaf on C x1¢ S. Then condition (2a) is satisfied by the definition of the functor. 
For condition (2b), let L/Iç be a field extension, and let i : Spec Iç -- Spec L 
be the associated morphism. Since i is fiat, it follows that (TrL).(1C X i)*/2 
/*(rg)./2. Now the condition follows sin«e ((1c x i)*/2)(-1)  (1c x i)*(/2(-1)). 
For condition (2c), we need to prove that the set of points {s} in S for which 
 d+g-(Spec k(s)) is open in S. We note that the image of a point s of S 
is in 
n. d+g-1 
in rcc/ lies in Spec A if and only if h°(C,, £(-1)) = 0 (Proposition 2.6). 
Now the condition follows by noting that the set in question is the inverse image 
r. d+g--1 
of Spec A under the morphisn S --, rlCc/ . 
Moreover, by the discussion above, it also follows that the canonical morphism 
ri. d+g--1 q-ld÷g--l¢ ç factors through Spec A. 
S -- rCc/k associated with all element of "C/k 
Note that the restriction of the uuiversal invertible sheaf to C x Spec A is an 
7-ld+a-l(Spec A). This follows from Proposition 2.6. This shows that 
element of c/ 
Spec A represents the functor "c/  [] 

V'e can now construct a natural transfornmtion 

, ,,{d+g -- 1 { 
' ç(s -) -- "c/ ,-)- 

Namely, to any elenmnt (gnd(g),¢) in 9d(C I, S), we assign the Ocxs-module 
(H°(S, ,ç) NOs Os[u, v]). To sec that  is a natural transformation, let f : T ---, S 
be a morphism of schemes and let (gnd(g), ) be an element of çd((/, S). Then 
we need to prove that the two invertible Ocx«T-modules OT NOs P'Sg and p,f*g 
are isomorphic. Now the statement is immediate, since the action of w is preserved 
by the natural O,.-module isomorphism between these sheaves 
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Since the fllllCtOr "C/k is representable, by sheafification of q, we get the 
natural transforlnatioi 

,,]_[ d-l- g -1 
 - r«,(ds,-)  ç(ds,-) ---. «/, (-). 
The isolnorphisln ill the above definition follows from Lemma 5.1. Now the functors 
_d+ 9-1 [ 
TePd(Ci,--) and C/k --) are represented by Spec Zk and Spec Ak, respectively. 
This follows from the fact that C i is an Azmnaya algebra over its center and 
Lemma 5.4. So the natural transformation induces a lnorphism of schelnes 

 : Spec Zk -- Spec Ak. 
In the following section, we will show that the lnorphisms r and 9 are inverses of 
each other. 

6. THE ISOMORPHISM THEOREM AND DESCENT ON 
Iii this section, we first prove that the lnorphisln  is an isomorphisln. Now 
consider the COnlposition 
Spec A/« - Spec Zk  Spec Ak, 
and denote it by 0. In the following proposition 7) denotes the restriction of the 
universal invertible sheaf to C× k Spec A. 
Proposition 6.1. The ovphis 0 is the identty ovphis on Spec 
"d+g-l(Spec Ak), it will be 
Pro@ Note that since 7 ) is the universal object in c/k 
sutïficient to prove that (le x 0)*7) is isolnorphic to 7) as a rigidified line bundle. 
Furthermore, by Remark 2.15 it is sutïficient to show that these two sheaves are 
isomorphic as line bundles. Note that the sheaf (1c x 0)*7) is isomorphic to the 
image under q) of (, cCd(7))) iii Tt«pd(CI, Spec A). But then the sheaf (1c x 0)*7) 
is isomorphic to (P® Ak[u. v]) ~, where P = H°(Spec A, (rrA).(7))) and w acts 
as u¢fl(Xl)+V¢fl(x2). Now ff'oin the construction of the morphism  (Proposition 4.2), 
the two actions of w on the graded (Ae lu, v, w]/(w d- f(, v)))-module P®  Ae lu, v] 
corresponding to the invertible sheaves (1c x 0)*7) and 7) on C xk Spec Ak are 
compatible. So the sheaves in question are isolnorphic. This shows that 0 is the 
identity morphism. [] 

Now we can prove the following theorem. 

Theorem 6.2. Let k be a field so that the curve C has a k-rational point. Then 
the morphism  is an isomorphism. 

Proof. We prove that the corresponding homomorphism op : Z  Ak is ail iso- 
morphism. By Proposition 4.8, it is sufficient to show that the map 9 is surjective. 
By Proposition 6.1, there exists a ring homomorphism  : A , Zk so that the 
composition 

is the identity homoniorphism. 
an isomorphism. 

A  Zk  Ak 
In particular,  is surjective. 

It follows that  is 
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6.1. The Galois descent on ç. In this section, we want to provethe isomorphism 
theorem of the last section without the assumption that the curve Chas a k-rational 
point. The idea is to prove that the 0 obtained earlier descends to a field over which 
fis defined. 
Let f be a binary form over a field k su«h that the characteristic of k does hot 
divide d and f does hot have repeated factors over an algebraic closure of k. Let 
k'/k be a filfite Galois extension (with Galois group (7) such that C(k') is non- 
empty. Here we inay assume that k' is a Galois extension of k by the assumption 
on char(k). Then we have the isomorphism 
o  Spec Ak,  Spcc Zk,. 
Recall that Spce Ak, descends to Spec Ak, which is the colnplelnent of the (-divisor 
ill F'ICc/k" al+g-1 Also, Npec Zk, descends to Sp«c Zk, which is the center of the Clifford 
algebra of f over k. 
Main Theorem. The morphism o descends fo k, and hence the center of the 
rcduced Clifford algebra is isomorphic to the coordinate ring of the complement of 
r. d+g-- 1 
the (-)-divisor in. 'Cc/ . 
Proof. Let x be a closed point of Spec Ak,, and let k(x) be its residue fiel& Let 
er Ç G. Then we have the following diagram: 
Sec t'(z) 
Spec Ak,  Spec Zk, 
Spec Ak,  Spec 
Now we prove that the above diagram is comnmtative. Let j denote the open 
n. al÷g--1 
ilnlnersion Spec Ak, '---, rmc,/,  We first show that 
(6.1) j oaoi = j O0 -100 O 00i. 
r. d÷g--1 - Xk ' 
Let 7r denote the structure inorphisln rCc,/, Spec k', and C' k(x) ° the 
fibred product of C' and Spec k(x) with the structure morphism for Spec k(:r) as 
7r o r o j o i. Also, let C' xk, k(x) denote the fibred produ«t as earlier with the 
structure morphism for Spec k(x) as 7r o j o i. We have the isolnorphism 
(6.2) C' x, k(x)  C' x, k(x) °. 
To prove (6.1), it is sufficient to prove that the line bundles on C  x, k(x) ' cor- 
responding to these morphisms are isomorphic. Sec Lemma 2.13. If T' denotes the 
r,. d÷g--1 
universal line bundle on C  x, rcc,/, , then the line bundle corresponding to the 
morphism on the left side of (6.1) is (id x (j o r o i))*T'. 
Consider the morphism on the right side of the above equation. To construct the 
corresponding line bundle, it is sufficient to consider the associated representation 
of t.he Clifford algebra. The required line bundle corresponds to the following 
composite morphism: 
C I ® k' idx__x_x CI ® k'  Ma(k(x)), 
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where r/i is the l'epresentat.ion corresponding to the morphislu i (or i o ç). But this 
line bundle is isomorphic to the pull-back of 7  under the composite morphisln 
C' x , l(x)  «L « C' x , k(x) «×(°)- C' x, P:«+--'c,/,  
Now consider the commutative diagram 

C' x, :(z) 
id × (joQ  
C I n. d+g--1 
Xk' lïlCc,/k, -- 

> C' x, k(x) ° 
id X (jo¢roi) 
 C I . d+g-1 
x Xk' VICc'/k' 

By the above argument and the conmmtativitv of this diagram, it follows that the 
two line bundles in question are isomorphic to 

(id× (jocroi))*T'- and (id× (jocroi))*(cr -l×cr-)*T ). 

By Lelmna 2.12, these two line bulMles are isomorphic. So (6.1) holds truc. 
In particular, i = (er - o ç- o er o ç) o i. Now consider the two autonlol'phisnls 
id an(l p = (er - o ç- o er o ç) of Spcc Aa,. They both are the identity ou closed 
points, and the morphisms induced on the residue fields are the saille. Let a  A«,. 
Then (tt(a) - a) is in anv maximal idem on Aa,. Siuce Ak, is of finite type over 
k' and is an integral dolnain, the intersection of all the maxilnal ideals is the zero 
ideal. So p(a) = a for any a  A,. This shows that p is the identity morphisln. 
The second part fi)llows ri'oin the (liscussion preceding the theoreln and Corollary 
2.7. [] 

We record an easy corollary of the maiu theorem. 

Corollary 6.3. The center Zk of the reduced Clifford algebra is an integrally closed 
Noetherian domain. 

Pro@ By the main theoreln, the k-scheme Spec Zk is isomorphic to the scheme 
n- d+--I - d+g--1 
Spe¢ Ak, which is the complement of the O-divisor in rlCc/k . Since r'Cc/k 
is a smooth, integral Noetherian scheme and Spec Ak is an open subscheme of 
; d+g-- 1 
Ptc/k , Spec Ak has these properties as well. In particular, A k is an integrally 
closed Noetherian donmin. So Zk is an integrally closed Noetheriau donmill. [] 

Remark 6.4. The Main Theorem 6.1 says that the reduced Clifford algebra can be 
viewed as a sheaf of Azunmya algebras on Spcc Ak. There are two natural questions 
n- d+g-- 1 
which arise. The first is, does this sheaf extend to rlCc/ as an AzulnoEva algebra? 
The second is if it does, then what is its Brauer class? These questions will be 
addressed in a forthcoming article [13]. 
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PROJECTIVE NORMALITY OF ABELIAN VARIETIES 

JAYA N. IYER 

ABSTRACT. We show that ample line bundles L on a g-dimensional simple 
abelian variety A, satisfying h°(A, L) > 2 g  g!, give projective normal embed- 
dings, for ail g >_ 1. 

1. INTRODUCTION 
Let A be an abelian variety of dimension g defined over the field of complex 
numbers and let L be an alnple line inmdle on A. Consider the associated rational 
nmp çr " A - pd-1 = pH0(L), where d = dimH°(A,L). Suppose L -- M n 
for some ample line bundle M on A. Then Koizumi has shown that L gives a 
projectively normal embedding if n >_ 3 (see [2]). 
When n -- 2, 0hbuchi (see [7]) bas shown the following. 
Theorem 1.1. Suppose lI is a symmetric ample line bundle on a g-dimensional 
abelian variety A. Then L = M 2 gives a projectivel normal embedding of A if and 
^ 
only if the origin 0 of A is not contained in Bsllll ® PI for any   A2 = {c  
Â  2c = 0}, where Â is the dual abelian variety of A, P is the Poincaré bundle 
on A × Â. P, = PIAx. for (  Â and Bsllll ® PI is the set of all base points of 
M®P. 
Suppose L ¢ ]I ' for any ample line bundle M on A and n > 1. When 9 = 2, 
Lazarsfeld (sec [4]) has shown that if 0r is birational onto its image, then 0r gives 
a projectively normal embedding, for d -- 7, 9, 11 and for d >_ 13. We showed that 
if the Neron Severi group NS(A) of A is N, generated by L and d _> 7, then 0r gives 
a projectively normal embedding (sec [11). 
In this article, we show 
Theorem 1.2. Suppose L is an ample line bundle on a g-dimensional simple 
abelian variety A. If d > 2 . g!, then L gives a projectively normal embeddzng, 
for all g >_ 1. (Her« d = dimH°(A, L)). 
Ve outline the proof of Theorem 1.2. 
For a polarized abelian variety (A, L), consider the multiplication maps 
p.  SymH°(A, L) - H°(A, L). 
By definition, L gives a projectively normal embedding if p« is surjective, for ail 
r _> 1. We first show that it suffices to show P2 is surjective. More precisely, we 
show that P2 surjective implies that the maps p. are surjective, for r _> 3 (see Prop. 
2.1). 
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Te prove the surjeclivity of the map P2 we consider a finite isogeny 
A/H, where H is a nmxirnal isotropic sui)grou t) of the fixed grou I)/£(L) of L. Then 
L descends dowll te a t)rincipal polarizati()n BI on /3. Let H (tenote the group of 
characters on H. By associating te a character \ 
on /3 one can identify // as a subgroup of the dual abelian variety Pic°(B) of/3. 
Thc homomorphism IbM " /3  Pic°(B), b , tBl ® BI -1 is an isomorphism and 
we denote H' = 
We then show that the surjectivity of the rnap P2 is equivalent to showing that 
the subgroup H' of B gcnerates th(, project ive space IPH°(B, M 2) and its translates 
lPH°(tBl2), where er  B is such that /,M(2er) = L, L  /;/, i.e., the images of 
points of H', under the rnorphisrn B 4);_ ]pHO(tkl.2) _ it3121, b  t;O+t*_+2oO 
(due to Wirtingcr), bave thcir lincar span as ItBl21. (tterc we assume that BI is 
symrnctric and that 0 is 
To sce this, wc show 

Proposition 1.3. Let E be an ample line bundle on a simple abelian variety Z of 
dimension 9 and consider the associatcd rational map Z  ]PH°(£). Then any 
finite subgroup G of Z of order strictly 9r«ater than h°(£)  9], 9enerates the Imear 
system llZH°(£). More precisely, the points Cz:(h) where h runs over all elements 
of G net m the base locus of £ span ]PH°(£) (.sec Prop. 3.4). 
We then apply Proposition 1.3 te £ = tBl 2 te obtain bounds as asserted for a 
polarized abelian variety (A. L) in Theorenl 1.2. 
Notation. The varieties considered in this article are defined over the complex 
numbers. 
Let/: be an ample line bundle on an abelian variety Z of dimension 9- 
1. The fixedgroup of £ is the group K(£) = {z 6 Z  £ _ t£}, t " Z ---, 
Z,x«z+x. 
2. The thetagroup of/:; is the group Ç(/;) = {(z,¢)  £  
3. The Weil form e t`  K(£) x I((£)  C* is the cornmutator map (x,y) , 
x'y'x'-ly '-1, for any lifts x', y'  Ç(£) of x, y  N(£). 
4. h°(£) = dirnH°(Z, £). 
5. If G is a finite subgroup of Z, then Card(G) = order(G). 

2. SURJECTIVITY OF THE MAPS Pe, r >_ 3 
Suppose/: is an ample line bundle on a 9-dirnensional abelian variet. A. Consider 
the mult iplicat ion maps 
H°(£) ®" - H°(£). for r > 2. 
The main result of this section is the following. 
Proposition 2.1. Suppose £ is an ample line bundle on an abelian vamety .4. If 
the multiplication map P2 is surjective, then p,. is surjective, for all r >_ 3. 
First, we recall 
Proposition 2.2. Suppose L and M are ample hne bundles on an. abelian vamety 
A. 
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1) The multiplication map 
ï t/°(t ® )® t/°(l ®  
is surjective, for any nonempty Zariski open subset U of Pic°(A). 
2) If the multiplication map H°(L) H°(M)  H°(L M) is surjective, then 
the multiplication maps 
(a) H°(L)  H°(M a) 
and 
(b) H°(L  a-l)  Ho(M  )  HO(L  M) 
are also surective, for  in some nonempt v Zariski ope subset U of Pic°(A). 
Proof. 1) See [3], 7.3.3. 
2) The proof is standard. 
Proof of Proposition 2.1. We prove by induction on r. Suppose the multiplication 
map p  H°(£)   H°(£ ') is surjective, for some r  2. 
Consider the composed multiplication map 
H0(£)+ 1 ld Ho(E )  HO(£) m, HO(£+). 
To see the surjectivity of the map p,+ = p« o (Id  p) we need to show that 
the map p. is surjective. 
Using Proposition 2.2 1), we cn write 
(,) H0().H0() =  HO().HO( -,).o(- ) 
CU 
for any nonempty Zariski open subset U of Pic°(A). 
Since P2 is surjective, by Proposition 2.2 2) (a), there exists a nonempty Zariski 
open subset U  of Pic°(A), such that for -  U , 

= H°(U+a). 

3. SURJECTIVITY OF THE MAP ff2 

Let Z be a g-dimensional abelian variety and let D be an ample divisor on Z . 
We denote/I : O(D) to be the ample line bundle on Z. Let (7 be a finite subgroup 
of Z. Consider the homomorphism bt : Z --- Pic°(Z), z - t*(ll) ® M -1. Let 
(7' C Pic°(Z) be the image of (7 under this homomorphism. Consider a finite 
subgroup J C Pic°(Z) and containing the subgroup (7'. Cnstruct an étale cover 
n : X --- Z corresponding to J, which is of degree equal to CrdJ. Let N = 
O(n-D) be the ample line bundle on X. 
Notice that if h  (7 ç K(Il), then t*ll - I, and this implies that D + h is 
linearly equivalent to D on Z. If bv  X  Pic°(X) is the map x - t*N ® 
N -1 and  : Pic°(z) --- Pic°(X) is the dual of the map 7r, then since ff(J) = 
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{0}, 71"--lG C K(N) = Kerbv (since bv ---- # o b^1 o r). This means that the 
divisors u-l(D + h) e IN[, for ail h e G. 
Coose the subgroup J such that N is base point free. (In fact, if J contains 
the subgroup of 3-torsion points of Pic°(Z) and G , then, by the above discussion, 
X[3I C K(N), where X[3I is the subgroup of 3-torsion points of X. This implies, by 
[3] 2.5.6, that N = K 3, for some mnple line bundle K on X and by a theorem of 
Lefschetz (see [31, 4.5.), N is very ample.) 
We will use the following. 
Lenma 3.1. Let I" be a variety and ç C Div(V) be an irreducible family of 
effective Cartier divisors Dt on V. Suppose Il" = tev Dt C t  and is nonempty 
and r = codim(ll'). Then there est divisors D), j = l, 2, ..., r, in ç that intersect 
r 
properly atd dimll" : dira i=1 D. 
Pro@ We use induction on j. Let D, D2, ..., Dj (j < r) be chosen in çsuch that 
they intersect properly in l'. Now write D1  D2  ...  D) = G1 U G2 U ... U 
where G, .... G, are irreducible components. Consider the closed subit i C ç 
parametrizing divisors that contain Gi for i = 1,2 .... ,s. (Note that II ¢ ç, 
otherwise Çi Ç II', which is hot possible since dimGi > dimIlç) Let U be the 
complelnent of " 
i= Il i in ç, which is nonempty since ç is irreducible. If Dj+I  
U, then D  ...  Dj  Dj+ has codilnension j + 1 (comlnunicated to us by A. 
Hirschowitz). 
Remark 3.2. Suppose D1, D2 .... , D are linearly equivalent effective divisors on a 
variety V, II" = i= Di and is nonempty and r = codim(lI'). If Pe denotes the 
span of the points D in the linear system [D[, then Il" = tee Dt. Hence, by 
Lemma 3.1, there are r divisors Dj  pk that intersect properly. 
With notation  above we have the following. 
Proposition 3.3. Let D be an ample divisor on a g-dimensional simple abelian 
variety Z. Let G be a finite subgroup of Z that is contained in D. Then Card(G)  
D  (whzch equals h°(O(D))  g, by the Riemann-Roch Theorem). 
Pro@ We prove this in several steps. 
Step 1:  reduce to the case when the divisors D and D+h, for ail h  G, are 
linearly equivaleut and O(D) is base point free. Indeed, by the above discussion, 
choose a triple (X, N. r), as above, corresponding to a subgroup J C Pic°(z) such 
that N is base point free and ¢(G) C J. This shows that the divisors -D 
and -l(D + h), Nr ail h  G, are linearly equivalent. Then we have a morphism 
çN : X  PH°(N). Since  is a finite morphism of degee equal to Card(J), 
by the projection Nrmula, one sees that deg(-W) = Card(J).deg(II'), for a 
subvariety IV of Z. Since (r-D)  = Card(J).D , if Card(r-G) 5 (-D) , then 
Card(G)  D . 
Step 2: We tan now assume that D is an ample divisor on X and that G C D 
is a finite subgroup such that D is linearly equivalent to D + h for all h  G and 
N = O(D) is base point free. Let }" = hea D + h and s = dira(Y). By Lemma 
3.2, Y Cj= Dj for soine 9 - s divisors Dj  INI that intersect properly. Now 
deg(Y) = [Y].[Dq (here deg(Y) = deg(S), where S C Y is of pure dimension s). 
Since Y C y= D we see that deg(Y) 5 D - In particular, when s = 0, since 
G C Y, we get Card(G) N D e- 
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Step 3: Supposethat s > 0. Let 1"= IUI)U...U), where}]-,1 j r, 
are the irreducible COlllpollents of }" such that s = dilll} 
degI'. Since )" is G-invariant, UhG 1 +  C }" and h deg()) + h) < degV, 
where G  {h G G "  + h = )} is a subgroup of G. Hence we get the 
Card(Gr ) .Dg" To 
inequalities Card().deg  degY  Da, i.e., Card(G)  deg 
complete out proof, we need to show that Card(G))  degY. 
Step 4: Now G C Stab(I) = {a G X  Y +a = )}. Observe that 
Stab(I) = ue I- g. Now for a point Y0  I, Stab(ti) = ()i-Y0)  t- 
y C (I - Y0)hea,e D + h- g. Let P = O - Y0) 
proceed to show that deg(Stab()i)) N dcg(P). This will be true if Stab(Ii) and P 
have the saine dimension. Now, we have 
 D+h-y =  t-y 
hG,y} yY 
=  ((y,   ... );.) - ) 
= ( ,-)( -)u...u( ;-). 
(To see the al»ove last equality: if   e () ) ...  )) - 
 U... U )ç, y E . Vi the translation map Y  Y U  U ... U , y  y + x and 
since  is irreducihle, x+y E ), for somej md for all y E , i.e., x E e )--y 
showing one way inclusion, the othcr inclusion being obvious.) 
We now sce that if j #  and «  e) )) - y, then ) +  « . If dim)- < 
dira,i, tlen this is absurd and so e ] - y is empty. If dim  dira]i, since 
] is of mimal dimension in Y, (lim)) = dim) and ) + x = . This implies 
that e  -y = e  + - = Stab0)+ . Hence 
P and Stab() are of equal dimension, say equal to m and deg(Stab0))  degP. 
Step 5:  proceed to show that deg(P)  deg(]). Consider t.he Poincaré 
line bmdle ç on X  Pic°(X). Let p and p denote the projections onto X and 
Pic°(x) respectively from X  Pic°(x). Consider the sheaf 
on Pic°(X). Since the vector spaces H°(N @ a) are of constant dimension for all 
a E Pic°(X), by Gruert's theorem, £ is  vector bundle on Pic°(X). Let (£) 
denote t he associated projective bundle on Pic ° (X). Considcr t ho natural norphism 
()  p7Nç. This is surjective, since on any fibre X 
which is globally generated (since N is globally generated) and £(a)  H°(N ). 
Hence this defines a morphism 5  X  Pic°(X)  (£). Let (£ denote the 
dual projective bundle over Pic°(X). In generl, the parameter space ç C (£ 
of the family (D + h - Y}eo,ei mv not form an irreducible variety (unless 
C = C), but we construct an irreducible subvariety  C (£ sucl that 
and heG,e) D + h - y = e D, where Dt denotes the divisor corresponding 
t t i («)(**). 
Step 6: Çonstruction of : 
Consider the subspace T of H°(X,N) spanned by sections s, h E C such 
that the divisor of s is D + . Consider the addition map 
X, (x, y)  x + y. Recall the skew-Pontryagin product of the sheaves x and 
N, N = (p ). (.« N) (see [8], p. 6a), where p(esp, p)  X  X  X 
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denotes the first (resp. second) projection. Then, by Grauert's theorem, N'Ox 
forms a vector bundle on X with fibres (NO.)  H°(tN). By [8], Remark 
1.2, N;Ox  N  Ox where N * Ox = m,(pi*N) is the Pontryagin product and 
by [5], p. 161, there are isomorphisms Ox @ H°(X,N)  NOx oe ¢ @ N 
(On " X  Pic°(X) is the isogeny x tN N-]). Çonsider the image F under 
f of the trivial subbmdle Ox  T in N$Ox. Then the fibre of F at x  X is the 
vector subspace of H°(tN) spmmed by the sections tsh whose divisor is D+h-, 
for b e G. Now (F) is a projective sul,b,,ndle of (E  N  (E (since 
N is a linc bundlc). Since  is irrcduciblc, the projective bundlc ?(F restricted 
to t is an irrcdu«ible subvaricty, and let ff be thc image of this irreducible variety 
in (E. Hencc ff is irreduciblc and, bv construction, if R  IGI,  , then 
¢ D + h -  C R and ff satisfics propcrty (**). 
Step 7: By Lcmma 3.1, therc cxist divisors DI, D2 ..... Dg-m   such that 
 ..  D-m C D  D2  ...  D_. This implies that deg(P)  deg(t - B0)), 
and by Ste t, 2 and Step 4. deg Stab(t)  deg  D  (since by Step 2, deg(t)  
deg(Y)  Dg). Siuçe X is simple, Stab(]) is zerdimensional and Gy, C Stab(t) 
implies that Card(G-,)  deg(). Hence by Step 3, Card(G) 5 D - This ends 
the proof. 
This is equivalent to the following. 
Proposition 3.4. Let  be an ample line bundle on a simple abelian vaetç Z and 
considev the associated vational map Z  H°(). Then anç finite subgvoup G of 
Z, of ovdev stctl9 gveatev than h°()  g, genevates H°(). More preciselç, the 
points ç(g) wheve g ns over all elements of G hot in the base locus of £ span 
We recall he following result, which we will need in the proof of Theorem 1.2. 
Proposition 3.5 (Wirtinger). Let (Z, (-)) be a pvincipall polavized abelian vavi- 
etç and  = 0(0) (heve 0 is assumed fo be a sçmmetc divisov). There is a 
nondegenevate innev pvoducl R : H°( 2)  H°( 2)  Ç (which is sçmmetc or 
skew-sçmmetvic depending on whethev the multiplicitg of the zevo element 0 on O. 
multoO, is even or odd) such that if R induces the isomohism 
(u0(c))  (u0(c).)  12ol, 
then the composed mohism 
R' 
z  (H°(C))  12OI 

is the mo-phism 

¢ : Z --- 1201, 
where Os ,s the translate of 0 by x on Z. 
Proof. See [6], Proposition, p. 335. [] 
Pvoof of Theovem 1.2. Consider a polarized simple abelian varietv (A L) of dimen- 
sion g such that h°(L) > 2 
Consider the nmltiplication map 
H°(L) ® H°(L) - HO(L2). 
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This map factors via 
Sym2H°(L)  H°(L2). 
Let H Ç K(L) be a maximal isotropic subgroup for the Weil forlll e L. Çonsider 
the isogenv A  B =  Thon L descends down to a principal polarization 
 H" 
M on B. We mav assume that M is symmetric, i.e., M  i'M, i(b) = -b,b 
B. Using the fact that .Oa = ¢U Lx, where L x denotes the degree 0 line 
bmldlc on B corresponding to the character y on H, by the projection fornmla, 
.L = ¢ M  L and .L  = xCu I   L. Hcnce we obtain the follo«ing 
decompositions: 
H°(L) = @ H°(M + Lx)H°(L ) = @ H°(M 
Vrite SymH°(L) = «'eO H°( M ç L')H°( M  Lx-x'-)" Con:Mer the 
nmlt iplica.tion maps 
 u0(  ,,).0(  ,«,_)  0(  ). 
Since p = e Px, it will suce to show thc su0ectivity of Px for each y  H. 
Since the pair (B, M) is principally polarized, the homolnorphism 
Pic°(B) is an isomorphism. Let H' = /() and 0  IM] be the unique symmetric 
divisor. 
Case 1: Suppose y is tïivial. 
Y see that the surjectivity of the lnap Ptriv is equivalent to showing that the 
reducible divisors 0 + O-h gencrate the lilea.r svstem ]I], for h  H'. Bv Propo- 
sition 3.5, using the morphism ¢  B  IM], this is the saine  saying that the 
image of the subgroup H' undcr the morphism 0 generates the projective space 
FH°(M). 
Case 2: Suppose y is nontrivial. 
First, notice that if b  B, then ai(b) = (2b). Let «  B be such that 
 («) = L, i.e., (2«) = L x. Hence the map Px is surjective if the reducible 
divimrs 0h + 0-h+« span the linear system [tM[ for h  H' = )(). Now if 
b  B, then 0 + 0_+« = (0«)_« + (0«)_+«, which is the image of the divisor 
0_« + 0_+« under the morphism [M[  [tM[ given by the translation map 
B B. Hencethe morphismç«  A  tM 2] isgivenas b 0+0_+«. 
This implies that Px is surjective if and onlv if the points in ¢«(H') generate the 
linear system 
Since the pair (A, L) is a simple polarized abelian variety with h°(L) = Card(H') 
> 2   g = h°(tM )  g, by Proposition 3.4, Px is sm'jective for ail y C . Hence, 
by Proposition 2.1, out proof is now complete. 
Remark 3.6. l) Suppose g = 1. Then any line bundle of degree strictly greater than 
2 on an ellipt.ic curve gives a project.ively normal embedding. Hence the bound is 
sharp. 
2) Suppose g = 2. If L  N , where N is an ample svmmetric line bundle 
with h°(N) = 2 on an abelian surface A. then it follows that h°(L) = 8 (in terres 
of "type" of an ample line bundle, N is of type (1,2) and hence L is of type 
(2,4) and h°(L) = 8). By [3], 10.1.4, N has 4 base points, say 
which are 4-torsion points on A and, moreover, 2xi C K(N) = Ker 'N where 
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N " .t  Pic°(A), o -, tN ® N -1 Let ci = bN(xi), fi,r i = 1.2, 3.4. Now the 
points a'i are base points for N, for i = 1,2,3,4, is e(luivalelt to saying that the 
origin 0 G A is a base point [kll" A" @ (ti, f()r i = 1, 2, 3.4. Also 2.ri  N(N) ilnplies 
that the points «i are 2-tol'si«m l)oints in Pi«o(A). Hence by Ohlnlchi's Theorom 
1.1. L does llOt give a projectively llOlllla.I cmbedding. So the bO/llld is sharp. 
3) Suppose g = 3. If L oe N 3, where N is a principal polarizatiol on an abelian 
threefold A, then h°(L) = 27. Bt bv Koizmni's Theorem, L gives a projectively 
nOl'lnal elnbedding. So the })Olllld iS 1101 sharp in this ce. 
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CLUSTERING OF CRITICAL POINTS 
IN LEFSCHETZ FIBRATIONS 
AND THE SYMPLECTIC SZPIRO INEQUALITY 

ç. BRAUNGARDT AND D. KOTSCHICI,. 

ABSTRACT. We prove upper bounds for the number of critical points in semi- 
stable syrnplectic Lefschetz fibrations. We also obtain a new lower bound for 
the number of nonseparating vanishing cycles in Lefschetz pencils and reprove 
the known Iower bounds for the cornmutator lengths of Dehn twists. 

1. INTRODUCTION 

It is a result of Szpiro [16] that iii a senlistable algebra.ic family of elliptic cllrves 
over CP 1 the number of critical points is boundid above by six times file nulnber 
of singular fibers, hl fact, Szpiro considered the arithnletic situation over a nunlber 
fiel& and the function field case was just a byproduct of these considerations, lqe- 
cently, Beauville [3] proved a generalisatiou of Szpiro's inequality to fibered surfaces, 
where both the base and the fiber have arbitrary geuus. 
Anlor5s et al. [1] gave a group-theoretic proof of Szpiro's inequality for semistable 
symplectic Lefschetz pencils, and this was extended to cert.ain hyperelliptic senti- 
stable symplectic Lefschetz pencils by Bogolnolov et al. [4]. Their result is that 
if ail the vanishing cycles are non-separating and the fibration has a topological 
section, then the nulnber of critical points is bounded above by 4h + 2 rimes the 
number of singular fibers, where h is file genus of a slnooth fiber. 
The purpose of this paper is to prove an inequality of Szpiro type for ail semistable 
symplectic Lefschetz fibrations of arbitrary base and fiber genus, without any as- 
sumption on the vanishing cycles and without the hyperelliptic assumption. In the 
case of pencils out result is that the number of critical points is bounded above 
by 6h(D - 1), where h is the genus of a smooth fiber and /9 is the nunlber of 
singular fibers. Sec Theorenl 15 and lemark 18 below. In genus one we once more 
recover the complex fuuction field version of Szpiro's theorem. Iii higher genus, 
our inequality is weaker than the one obtained by Beauville [3] in the algebraic 
situation. 
The point of the symplectic Szpiro inequality is that while oue can alwa.vs per- 
turb a Lefschetz fibration in the neighbourhood of a siugular fiber so as to make it 
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injective on its critical set, there are global obstructions to the clusteriug or con- 
centration of critical points in a fiber. These ol)st.ructions are essentially the ones 
discovered by Eu(lo and Kotschick [6] in their proof that the commutator lengths 
of powers of Dehn twists have linear growth. 
In Section 3 we generalise the main result of [6] to relatively minimal Lefschetz 
fibrations which ueed hot be injective on their critical sers, and then deduce a Szpiro 
ilwquality for fibrations over bases of positive genus from this geueralisation. We 
then rederive the lower bounds for the commutator lengths of Dehn twists proved 
tu [6], [8]. There the argulnelt with the Kneser inequality from [9] was combined 
with the observation that iteratiou of Dehn twists makes the signatures of Lef- 
schetz fibratiolls more and more uegative. This is clear for separating Dehn twists, 
lmt also works for uonseparating Dehn twists by using a handle decomposition, 
cf. [8]. Here we give a treatm(,nt that avoids handle decompositions and derives 
the growth of the negative d(,finite part of the intersection form from the purely 
homological Proposition 5 which is staudard in algebraic geolnetry but holds also 
ill out sylnplectic setup. This luakes the argument self-contained: in particular, it 
is iudcpeudent of signa.ture calculatioliS for Lefschetz fibrations carried out. ad hoc 
or lv using the Meyer signature cocvcle. 
In Section 4 we gencralise results of Li [10] and Stipsicz [14], [15] to relatively 
nfinimal symplectic Lefschetz fibratiols which need hot be injective on their crit- 
ical sers, and then derive a Szpiro inequality ff'oto these generalisations. We also 
prove that the mlmber n of nouseparating vanishing cycles in a Lefschetz pencil of 
genus h is no less thal (Sh - 3). The best previously knov«n bound was n _> h, 
cf. Remark 22. Here too our argument is independent of any signature calculations. 

For 
Let 
h >1 
fibers, 
fibers. 

2. SEM[STABLE SYMPLECTIC LEFSCHETZ FIBRATIONS 

definitions and background on differeutiable Lefschetz fibrations we refer 

f: X -- B be an oriented Lefschetz fibratiou with base genus g, fiber genus 
 with n nonseparating and s separating vanishing cycles. We denote bv 
+ s the total lmmber of vanishing cycles aud by D the lmmber of singular 
so that k _> D. We denote by N the total lnlmber of compoueuts of singular 
Note that for h = 1 we have k = N. 

We shall assume throughout that the total space X is a symplectic manifold in 
such a way that the fibers are symplectic submanifolds. Bv a theorem of Gompf, 
cf. [7], this is no restriction if h _> 2. 
The Euler characteristic of a Lefschetz fibration is given by 
(1) ,(X) = 4( - 1)( - 1) +  . 
The following is elementary: 
Lemma 1. If K denotes the canonical class of an almost complex structure asso- 
ciated with the symplectic structure, then 
h  = 5,((X) - 6 + 6bl (X) - 6b-(X) 
(2) 
= 20(9- 1)(h- 1)+5k-6+6bl(X)-6b(X). 
Definition 2. A symplectic Lefschetz fibration f: X -- B is called semistable if 
every 2-sphere component of a singular fiber contains at least two critical points. 
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The llalne COllle8 frein the fact that if we choose a I:(ienannian metric on X and 
consider the induced conformal structures on the fibers, these become semistable 
algebraic curves if and only if the above topologi«al condition is satisfied. 

Lemma 3. A symplectic Lefs«hetz fibrotion is semistable if and only if it is rela- 
tively minimal. 

Pro@ Assume it is semistable raid suppose that a singular fiber has more than 
one colllpOllellt. Let S be ail irreducible conlpollent. If F denotes the class of the 
generic fiber, then F- S = 1}. Thus, 
S'2 = S-(S- F) _< -1 
since the singular fiber is connected, and different components intersect positively 
unless they are disjoint. If S. (S- F) --- -1, then by the definition of semistability 
S is hot an embedded sphere. Thus there is no embedded sphere of self-intersection 
-1 in any fiber. 
Conversely, if the fibration is relativelv nfinimal, then by the saine calculation as 
above, there Call |)e 11o spheri«al COlllpollellt containing onlv one critical point. [] 

A variation of this calculation shows thc following: 

Lemma 4. Let X be a Lefschetz fibration and Fo a singular fiber. Then H2(X) 
contains a negative definite subspace for the intersection form spanned freelg by the 
classes of all the components of Fo but one. 

Pro@ In the case of algebraic surfaces, this is well knov«n as Zariski's lenmm: sec 
e. g. [2]. The argument given there goes through in the symplectic situation because 
the components of singular fibers that we have in the statement intersect each other 
positively if they are net disjoint. [] 

Proposition 5. Let X be any Lefschetz fibration. Then 
(3) b(X) >_ I + N- D . 
Pro@ The negative definite subspa.ces of H.2(X) obtained by applying the preced- 
ing lemma te the different singular fibers are mutually orthogonal. Thus the direct. 
sure of all these subspaces is nega.tive definite of dimension N - D, and is still 
orthogona.1 te the class of a generic fiber, whi«h has zero self-intersection. [] 
We shall also need the following estinmtes for the nunlber of components of 
singular fibers: 
Proposition 6. For every Lefschetz fibration we bave 
(4) N >_ s+D, 

(5) N >_ k - (h- )D. 

Pro@ The first inequality is immediate from the definition. 
Te prove (5), let N be the union of the singular fibers. We tan calcula.te the 
Euler «haracteristic of N by comparing the singular fibers te a generic fiber F, 

\() = D((F) + k = -2(h - 1)D + k, 
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or bv sumnfing over ail components CI ..... CN of the singular fibres, 
N 
\(r) =  x(cd - k   - k 
i=1 
Thus we have N  k - (h - 1)D.  
Remark 7. For fibrations with only stable fibers we have k  3(h - 1)D and 
N  2(h- 1)D for purely topological reasons. 

3. LEFSCHETZ FIBRATIONS OVER BASES OF POSITIVE GENUS 
In this section we prove the main technical result about relatively minimal Lef- 
schetz fibrations and derive inequalities of Szpiro type under the sumption that 
the base has positive genus. 
Theorem 8. Let X be a connected smooth closed oriented 4-manifold and f" X  
B a relatively minimal symplectic Lefschetz fibration with fiber genus   1 and 
base genus g  1 havig k vanshng cycles, D sigular fibers and N ieducble 
componcnts of sngular fibers. Then 
(6) 5k +6(3h- 1)(g- 1)  6(N- D) . 
Pro@ Since X is SUlned to be rclatively lninimal, the positivity of the be genus 
implies that X is lninimal and hot rulcd, because any pseudo-holomorphic sphere 
in X would hae to be contailmd in a fiber. Thus Liu's extension [11] of Taubes's 
results [17] ilnplies K 2 k 0, which we can write  
1 
(7) b(.¥) 2 g(b(A') + 4b, (A')-4) . 
Using (3) and bi(X) OE 2g  2, we obtain 
1 N . 
b(X) l+g(-D) 
Since the claire (6) is trivial for N = D, we may sume N - D  1, and therefore 
(x)  2. 
Since X is mininml with bf(X)  2, we tan use the result of Taubes [171 to 
obtain a symplectically embedded surface N C X representing the canonical cls 
K of X. Il nv be disconnected, bnt because X is nfinimal,  h no spherical 
COlnponent. hl the gument below we will tacitly sunle that it is connected. In 
the general case the saine argulnent works by sulmning over the components. 
The genus of  is given by the adjunction formula g(£) - 1 = K . The fibration 
f induces a smooth map  + B of degree d equal to the algebraic intersection 
nnmber of  with a fiber. This is calculated from the adjunction fornmla applied 
to a smooth fiber F, which is a symplectic subnmnifold: 
(8) d = -F = K-F = 2h- 2. 
Thus Kneser's inequality g(E) - 1  Id](g(B) - 1) gives 
tç 2  2(h- 1)(9- 1). 
Combining this with (2) and estinmting K  from above using b (X) OE 1 + N - D 
and bi (X)  2g + 2h we obtain 
6(N-D) 18(h-1)(9-1)-12+12 9+12h+5k. 
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Pulling the fibration back to large degree covers of the base B and applying the 
al)ove inequality, we finally ot)tain (6). [] 

Corollary 9. Let X be a connected smooth closed orie.ted 4-mawifold and f" X -- 
B a relatively minimal syrople«tic Lefs'chetz fibrotion with fiber genus h >_ 1 and 
base genus g OE 1 baving s separatirg a.d t .onseparating vanishing cycles, D 
singular jïbers and N irreducible compone.ts of singular fibers. Then the .following 
inequalities hold: 

(9) 

s < 6(31, - 1)(g - 1) + 5., 

(10) 

t. < 631, - 1)(g - 1) + 61,D, 

(11) _N < 6(31,- 1)(g - 1) + (5h + 1)D . 
Proof. The first claire follows from (6) using N >_ s + D. The second and third 
claire follow similarly using ]: < N + (h - 1)D. [] 
Remark 10. The inequality (9) was originally proved by Endo and Kotschick [6] 
un(ler the assmnption k = D. 
Remark 11. Note that for h -- 1 we bave s = 0 and k = N. Fronl (10) or (11) we 
obtain 
k=N< 12(9-1)+6D, 
which is a generalisation of the Szpiro inequality to semistable sylnplectic Lefschetz 
fibrations over bases of positive gelmS. For h _> 2, either (10) or ( 11 ) tan be regarded 
as a Szpiro-type inequality. 
We now apply the previous discussion to give a new proof for the known lower 
bounds for the commutator lengths of powers of Dehn twists in mapping class 
groups. 
Theorem 12. Let a be a homotopically nontrivial siple closed curve on a surface 
F of genus h >_ 2. and let t be the corresponding Dehn twist. Suppose that t with 
k > 0 can be written as a product of g commutators. Then 
k 
(12) g _> 1 + 
6(3h - 1) " 
Proof. Consider the standard holomorphic Lefschetz fibra.tion over the 2-disk D 2 
with precisely one singular fiber Fo with vanishing cycle a. Pulling back under the 
base change z - z  and taking the minimal resolution, we obtain a holomorphic 
Lefschetz fibration over D 2 with onlv one singular fiber having k vanishing cycles 
that are parallel copies of a. The monodromy of this fibration around the boundary 
of the disk is t. If this can be expressed as a product of g commutators, then we 
can find a smooth surface bundle with fiber F over a surface of genus g with one 
boundarv component and the saine restriction to the boundary. Let X be the 
Lefschetz fibration over the closed surface B of genus g obtained by gluing t.ogether 
the two fibrations along their common boundary. 
By construction, X is symplectic and relatively nfininml, so that we can apply 
Theoren18. We have D = 1 and N = k+ 1 ifa is separating and N = k ira 
is nonseparating. Theorem 8 gives k _< 6(3h - 1)(g - 1) + c with c = 0 or c = 6 
depending on whether a is separating or hot. In the latter case, by pulling back the 
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fibra.tion to lin'ge degree coverings of the base, we also obtain k <_ 6(3h - 1)(g- 1) 
as claimed. [] 

Remork 13. The incqnality (12) was originally proved by Endo and Kotschick [6] 
under the assmnption that (! is separating. In the context of Lefschetz fibrations 
with k = D they used the separating assumption to conclude bf(X) >_ k + 1 
whenever  ocçurs k tilnes as a vanishing cycle, since every separating vanishing 
cvcle lnakes a negative contributiol to the signature. Korkmaz [8] then observed 
that using a handle deCOlllpositiol one still gets b(X) >_ k in the nonseparating 
case, so that the argulnelfl goes through. Phrased as above, the proof works directly 
fir |tll cases, since the required lower louud for b (X) arises from the homological 
argUllWlt in Propositiol 5. 

4. LEFS('ttETZ PENCILS 
In this se«tim we consider the case of pencils, i.e., Lefschetz fibrations over the 
2-sphere. \Ve shall assnlne that ont pencils are nontrivial, meaning that they have 
at least one critical point each. 
As in [10], [15], the case of ruled surfaces ha,, to be considered separa.tely. 
Proposition 14. Suppo.se X is the blow.up in b points of a 2-sphere bdle over 
a surf«ce of 9enus a, and hat X admits a nontrivial relativelg minimal sgrnplectic 
Lefschetz pecil with fiber genus h >_ 1 having s separating and n nonseparatin 9 
vanishi 9 cycles, D singular fibers ad N irreducible coupoents of singular fibers. 
Then the followin9 inequalities hold: 
3 
(13) k > 2 - 2 + (N - D), 

1 
(14) n >2h-2+(N-D). 
Pro@ By the assumptions OIl .¥, vce have b2 + = 1. b- = 1 + b and bl = 2a. 
Stipsicz [14] proved that for a nontrivial relativelv lnilfimal Lefschetz pencil, 
K 2 _> 4(1 - h). His proof was written under the assulnption that Lefschetz pencils 
are injective on their critical sers, but this can be achieved by perturbation, and 
the inequality involves only topological invariants which do hot change under per- 
turbation (Ulflike D and N); so the inequality is truc in our case. Substituting the 
above nuinbers into it, we obtain 
. lb 
(15) 4a<_ ")+2h- . 
Computing the Euler characteristic of X in two different ways, we see that 
4(1 - a) + b = \(X) = 4(1 - h) + k, and so 
k 
=b+4h-4a>2h-2+ =2h-2+2(b--1) >2b-2+8 ' 
where we have used first (15) and then Proposition 5. Thus we have proved the 
first claire. The second one follows frolu the first using Proposition 6. [] 

In general we have: 

Theorem 15. Let X be a connected smooth closed oriented 4-manzfold and f " X  
S 2 a nontrivial relatively minimal symplectic Lefschetz pencil with fiber genus h >_ 1 
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havi,g s separatiTg and n nonseparating vanishing cycles, D singular fibers and N 
irreducible cornponents of singular fibers. Then the following inequalities ]old: 
(16) 5 _> 6h + 6(V - D) , 

(17) 5 _> 6h + s , 

(18) k <_ 6h(D- 1) , 

(19) N _< (5h + 1)(D- 1) - (h - 1) . 
Pro@ First we fiber sllnl \" with a genus ] bundle over the 2-torus and apply (9) 
to the resulting Lefschetz fibration. This shows that, since X is nontrivial, we lnust 
have n > 0. Therefore bi(X) _< 2h - 1. 
Assuming first that X is hot rational or ruled, we would like to use Taubes's 
result [17] as extended by Liu [11] to obtain Iç 2 >_ O. The probleln with this 
argulnent is that in the case of base gelms zero, relative milfimality does hot imply 
minilnality, so that Taubes's result is not available. However, if X is hot ratiolml 
or ruled, we have Li's inequality [10] 
(20) K 2 >_ 2 - 2h . 
Li's argument assulnes that the Lefschetz pencil is injective on its critical set, but 
this can be achieved by perturbation without affecting the inequality. 
Using this, we proceed as in the ploof of Theoreln 8. Colnbinilg (20) with (2) 
we obtain (16) by using g = 0, b(X) _< 2h - 1 and b(X) >_ 1 4- N - D. 
It remains to deal with the case that X is a (blowup of a) ruled surface. If h >_ 3, 
then (16) follows from (13). 
If h = 2, suppose we have a fibration satis'ing (13) but failing (16). Then we 
conclude that k = 2, giving \(X) -- 4(1-h)+k = -2. On the other hand, in 
  which gives a < 1. 
the notation of Proposition 14 we have a _< ½h +  - gb, _ 
conclude that \(X) = 4(1 - a) + b _> 0, which is a contradiction. 
If h = 1 for a fibration X satisfying (13) but failing (16), then we conclude that 
k _< 3 and therefore ,\(X) =/," _< 3. As above we also obtain a _< 1. If a = 0, then 
 ((X) = 4, which is a contradiction. If a = 1, then (15) gives b = 0 and therefore 
k = x(X) = 0. But then the fibration is trivial. 
Thus (16) is proved in all cases. Once we have (16), the other inequalities follow 
as in the proof of Corollary 9. [] 
Remark 16. For non-ruled total spaces, the inequality (17) was proved by Li [10] 
under the assumption k = D. 
Remark 17. If k ¢ 0, we conclude that D _> 2 from (18). Thus, the critical 
points of a nontrivial Lefschetz pencil tan never be concentrated in a single fiber, 
compare [12]. 
Remark 18. Note that for h = 1 we have k = N and both (18) and (19) reprove 
Szpiro's inequality [16] and give extensions to pencils with h > 1. Concerning (18), 
note that Bogomolov et al. [4] proved that/," <_ (41 + 2)D for hyperelliptic fibrations 
under the additional aSSUlnption that all vanishing cycles are nonseparating and 
that the fibration admits a topological section. 
Remark 19. In the proof of Theorem 15 we have used b (X) _< 25.- 1. If we actuall,v 
know the first Betti nulnber, then we obtain better inequalities. 
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The following heorem gives new bomMs on the nmnber of nonseparating van- 
ishing cycles in Lefschotz pencils. 
Theoreln 20. Let X be a connected smooth closed oriented 4-manifold ad f" X  
S  a nontrivial relotively minimal symplectic L@chetz pen«il with fiber genus h  1 
havin9 s separatin9 and n nonscpa.rotin9 vanishing cycles. D singular fibers and N 
irreducible conpon¢'nfs of singulr fibers. If X is o rafional or ruled, then he 
followin, g inequalities hold: 
(21) 5kkSh-3+5(N-D) , 

(22) 5n _> 8h- 3 . 
Pro@ The assmnpti()n that fis relatively minimal again does hot imply that X is 
minimal, in which case we woul(l have h   0. However, h -  0 can be rewritten 
(3) b - 4, + 4  . 
Blowing up or down does hot chauge the left-hand side of this inequality; so, re- 
gardless of whether X is miuimal or hot, the fact that it is hot rational or ruled 
implies 
(24)  + 
oh 2 (X) - 4bi(X) + 4 OE 0 
because of [7] and [1]. çsing (1) and 
(X) = b(X) - b; (X) = X(X) -  + b(X) - b(X) , 
we obtain 
(25) 5k  20h - 4 - 6b(X) + 5b. 
As in the proof of Theorem 15 we bave b (X)  2h - 1. From Proposition 5 we 
have b(X)  N- D + 1. Thus (25) gives (21). 
Using N - D  s, we obtain (22) from (21).  
Fina.lly, we extend (22) to ail LeNchet.z pencils: 
Theorem 21. If f: X  S 2 is a nontrivial L@chetz pencil of geus   1. then 
f bas at least (Sh - 3) onseparating vanishing cgcles. 
Pro@ Clearly it suffices to prove the relaively minimal cse. If X is hot rational 
or ruled, then the result is part of the previous Theorem, cf. (22). If X is rational 
or ruled, we have (14), which is enough as loug    4. On the other hand. if 
h  2. then (17) implies the claire. Thus it renmins to deal with the case b = 3 for 
ruled manifolds. 
If h = 3 and X is the blowup of a ruled surfe satis[ving (14) but Niling 
  (Sh- 3), then again s N N- D = 0, and k = n = 4. The smne arguments 
 in the proof of Theorem 15 above then give b = 0 and a = 2. Thus we have a 
ruled surface without any blowups over a geuus 2 surface, in which the generic fiber 
F of the LeNchetz fibration represents a homology cls of zero self-intersection. 
Since F is a symplectic subnmnifold, the proof of the Thom conjecture implies that 
F h minimal genus in its homology cls. We now show that this leads to a 
tout radiction. 
If X is the product ruled surface S 2 x E2, t.hen the homology cluses of zero self- 
intersection are the multiples of the two factors. The multiples of the first ftor 
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are all represented by spheres, so F could only be a nmltiple of the second factor. 
Since it has larger genus than the second factor, it would represent d times the 
second factor with [d] > 2. But the projection to the second factor induces a map 
of degree d from F and from the normalisation of a siugular fiber to OE» Since the 
normalisation of a singular fiber bas genus _< 2, it does hot nlap to OE2 with degree 
>2. 
For the nontrivial ruled surface, F can also hot be a multiple of the class of the 
2-sphere in the ruling, and nmst therefore be a class that has intersection number 
d -J: 0, 4-1 with the class of the sphere in the ruling. Thus it maps to the base 
surface of genus 2 with degree _> 2, and we obtain the saine contradiction as in the 
product case. [] 

 " ' l(8h -4) 
Remavk 22. Theorem 6.1 of Stlpsmz s paper [15] gives the lower bound g 
for the total number of critical points in nontrivial Lefschetz pencils (which are 
injective on their critical sets). However, the proof given there does uot seem 
to cover the case of ruled surfaces with fibrations of small liber genus, because 
it appeals to Theorem 1.1 which gives nothing for fiber genus <_ 5, for example. 
More importantly, Stipsicz's argument appeals to Theoreln 1.4 of that paper, whose 
proof is incomplet.e, because it uses a stat.euent about the fundamental group of a 
Lefschetz fibration which is hot known to be truc. In fact, it is false if one considers 
achiral Lefschetz fibrations; compare [1], p. 503. 
In any case, the important difference between the result of Stipsicz and ours is 
the fact that we obtain a bound growing with -h for the number of nonseparating 
vanishing cycles, rther than the total uumber of separating and nonseparating 
ones. In the nonseparating case the best previous result is that of Li [10], which is 
n> h. 
Remark 23. Cmbining Theorems 15 and 21, the nmnber of nonseparating vanish- 
ing cycles in a Lefschetz pencil is bounded below by 
(26) 5n _ fs + (8- 2t)h + 3(t- 1) 

for ail t  [0, 1]. 

5. FINAL COMMENTS 
As in [9], [6], the arguments of this paper rely on the work of Taubes [17] in 
Seiberg-Witten theory, showing that for a nfinimal symplectic 4-manifold with 
b + > 1, the canonical class is represented by a symplectically embedded surface 
without spherical components. Recently, the methods pioneered in Donaldson's 
work on Lefschetz pencils have been applied successfully to reprove Taubes's result. 
Donaldson and Smith [5] did this under the additional assumption b +-b > 1. This 
is hot sucient for out purposes, but, as explained by Smith at the end of [13], the 
arguments of [5] can be pushed to cover all cases where b + > 2. 
This means that as long as we work with nmnifolds with b + > 2, out results 
can be proved independently of gauge theory. In Section 3 we appealed to Liu's 
work [11] in gauge theory for the case b + = 1, but this can be avoided. Therefore, 
out results on fibrations over bases of positive genus do hot require any input from 
gauge theory. In the case where the base genus is zero, it is possible that b + = 1 or 
2, but if we exclude those cases, then the proofs can be based on [5], [13] instead 
of [17], [111. 
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ON MEASURES OF MAXIMAL AND FULL DIMENSION 
FOR POLYNOMIAL AUTOMORPHISMS OF C 2 

CHRISTIAN V'OLF 

ABSTRACT. For a hvperbolic polynomial automorphism of C 2, we show the 
existence of a measure of naximal dimension and identify the conditions under 
which a rneasure of full dimension exists. 

1. INTRODUCTION 
Let g be a hyperbolic polynonlial autonlolp|lisln of C2. For .4 C C2 we denote 
by dilnH A the Hausdorff dimension of A. Let u Se ail invariant Borel probabilitv 
measure. We define the Hausdorff dimension of oe by 
(1) dinlH(oe) = inf{dilnH A: Oe(A) = 1}. 
We define the quantity ri(g) by 
(2) ((g) = sup{dimH(u)}, 
where the supremum is taken over all ergodic invariant Borel probability measures 
with positive entropy. 1 This quantity was introduced bv Denker and Urbanski 
[DU] in t.he context of rational lnaI)s on the Pdelnalln sphere. They called it the 
dynamical dimension of the nlap. 
It is easy to sec that the support of each measure considered in (2) is contained 
in the Julia set J (see Section 2 for t|le definition). We denote by ll(J, g]j) the set 
of ail ergodic invariant Borel probability measures supported on J. 
If a measure  G M(J, gIJ) attains the supremum in (2), that is, 
(3) dim(u)=5(g), 
we say that u is a measure of maxinml dimeusion for g. 
McCluskey and Malming [MM] gave a heuristic argulneut for the existence of 
a measure of maximal dimension in the case of Axioln A surface diffeomorphislns. 
However, it was hot known until this paper v«hether this argument can be extended 
to a rigorous proof (sec the renmrks after Theorem 5.2 for more details). 2 
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Ill this Imper we study the existence of a measure of maximal dimension for 
hyperbolic polynomial automorphisms of C 2. Our lnain result is the following (see 
Theorems 5.1, 5.2 and Corollary 5.4). 
Theorem 1.1. Let 9 be a hyperbolic polynorniai autornorphisrn of Ç 2. Then there 
exists a rneasure of nmzirnai dimension for 9- The set of rneasures of maximal 
dirnensioTt is fitite. 
Thc 1)roof of this 'dmorem uscs 'he thcory of thermodynamic formalism. Other 
crucial ingrcdients are Young's formula [Y], which relates the dimension of a mea- 
sure to entropy and Lyapmlov exponcnts, and (timension-theoretie results pertain- 
ing to 'he dynanfics of polynomial automorphisms of C 2 (see [Wol], [Wo2]). The 
key idca is to extract a one-paramcter fmnily of potentials and to consider the cor- 
rest)omling family of equilibrium lncasurcs, ,Ve show that a measure of maxinml 
dilnensiol neccssarily belongs to this falnily of equilibrimn measurcs. Furthermore, 
if a measure u maximizcs Hausd(wff dimension among 'hese equilibrium measures, 
then u is a meesure of maximal dimension. 
We say that u  AI(J, g[,) is a measurc of full dimension if 
(4) diln(u) = dimH J. 
We prove in Corollary 3.5 that there exists at most one measure of full dimension. 
The next result gives a classification for the existençe of a measure of full dimension 
(see Theorems 4.1 and 4.2). 
Theorem 1.2. Let 9 be a hyperbolic polynornial automorphism of C 2. Then 
i) If9 is volume-preservin9, then there exists a unique rneasure of fuli dimen- 
sion for 9. 
il) If9 does ot preserve volume, atd if9 admits a rneasure of full dimension, 
then this measure is the unique rneasure of maximal entropy. 
51 the volume-preserving case the existence of a measure of full dimension has 
already been shown b,v Friedland and O«hs [FO]. We provide an alternative proof 
for this result in Theorem 4.1. 
Theorem 1.2 indicates that in the case of non-vohmm-preserving maps, the ex- 
istence of a measure of full dinlension seelns to be a rare phenonlenon. Indeed, we 
show iii Theorem 4.2 that the existence of such a measure is equivalent to the fact 
that the nmltipliers of ail saddle points are of the saine modulus (see equation (28) 
for the precise statement). If this condition would be satisfied, then the parameter 
defining the map 9 would provide solutions of a countable infinite set of algebraic 
equations, lom this point of view, such an example seems very likely to not exist. 
Iii Section 4 we observe that there exists a dense open subset of hyperbolic 
parameter space for which no measure of fifll dimension exists. This implies that 
(5) 5(g) < dimH J 
holds for t hese parameters. 
In the last part of this paper, we analyze the dependence of 3(g) on the parameter 
of the mapping. More precisely, we prove the following result (see Theorems 6.1 
and 6.2). 
Theorem 1.3. Let  - g, be a hoiornorphic farnily of hyperbolic polynomiai au- 
tornorphisms of C 2. Then  -, 3(g) is continuous ad plurisubharmonic. 
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This paper is organized as follows. In Section 2 we prescrit the basic definitions 
and lotation, and also list the standing assumptiols of the paper. In Section 3 
we introduce elelnents froin dilnension theory for hyperbolic polynonfial automor- 
phislnS of C 2 and provide the tools for the analysis of the existence of measures of 
maximal and full dillmnsion. Section 4 is devoted to the almlvsis of the exist.ence 
of a lneasure of full dilnension. The exist.ence of a lneasure of maximal dilnension 
is proved in Section 5. Finally, we studv in Section 6 the dependellce of 6(9) on 
parameters. 
It would be inleresling to understald whether, or at least under which conditions, 
a Ulfiquelmss resull for the lneasure of maxilnal dimension holds. A partial answer 
to this question is given in Corollary 3.5 where a UliquelmSS result is shown in the 
case when a lneasure of fidl dilnension exist.s. 

2. NOTATION AND PRELIMINARIES 
Let g be a polynomial alltOlnOrl)hiSll of Ç2. We denote by deg (g) the lllaXilllinn 
of the polylomial degree of the COlnpOlmnts of g. The dvnalni«al degree of 9 is 
defilmd by 
d= lira (deg(g)) t/n 
(sec [BS2], [FMI). We are interested in l,ontrivial dynamics, which occurs if and only 
if d > 1. Friedland and IIilnor showed iii [FM] that every polynomial automorphisln 
of Ç2 with nontrivial dynamics is conjugate to a mapping of 1 he fOl'lll ff = fil O...Off,n , 
where each 9, is a generalized Hénon lnapping. This means that g, has the form 
(6) gi(z, W) = (W. Pi(w) q- az). 
where Pi is a COlnplex polynomial of degree di >_ 2 and ai is a nonzero colnplex 
lmlnber. The dvnamical degree d of such a map g is equal fo dl'...-dru and therefore 
coincides with the polynolnial degree of g. 
For g we define K + as the set of points in (2 with bounded forward/backward 
orbits, K = K + fqK-, ,_/4- = OK4- and .1 = J+ç/J-. \Ve lefer t.o J+ as the 
positive/negative Julia set of g and call .1 the .hflia set of g. The sels K and J are 
compact. 
Note that the function o = det Dg is constant in Ç2. Therefore, we can restrict 
our considerations to the vohlme-decreasing case (lai < 1), and to the vohnne- 
preserving case (lai -- 1), because in the volulne-increasing case (lai > 1), we can 
consider g-l. 
As pointed out in the introduction we assunm in this paper that g is a hy- 
perbolic mapping. This means that there exists a continuous invariant splitting 
TaC2 = E  @ E s such that Dg[F, is uniformly expanding and DglF, is uniformlv 
contracting. Hyperbolicity implies that we can associate with each point p G J its 
local unstable/stable manifold l l''*/* (p). Moreover, g is ai1 Axiom A diffeomorphisln 
(see [BS1] for more details). 
Standing Assumptions. We now list several conditions which will be assumed 
throughout the entire paper: 
i) g is a polynomial automorphism of Ç2 with dynamical degree d > 1: 
ii) g is hyperbolic; 
iii) g is volulne-nol-increasing. 
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"We recall that assumption iii) is actually hot a restriction since we can also consider 
9 -1 (sec above). 

3. ELEMENTS FROM DIMENSION THEORY 
In this section we introduce elemeuts from dimension theory for hyperbolic poly- 
nonial automorphisns of C  and provide the tools for the analysis of mesures of 
maximal and fltll dimension. 
We start by introducing Lyapmmv exponents. Let u  M(J. g]j). The Julia set 
J is a hyperbolic set of index 1 (sec [BS1]). Thcrefore, bv the multiplicative ergodic 
theorem of Oseledec, there are Lyapuuov cxpoucnts A < 0 < A with respect to 
u (sec e.g. [KH]). Iu particular, u is a hypcrbolic measurc. We define the quantity 
Similar to what w donc for the measure of maximal entrow in [BS3], the pos- 
itive Lyalmuov exponeut A coincides with A(u). Since g h constaut Jacobian 
determinant a, the negative Lyapmlov expouent A 2 is given by -A(e) + log lai. In 
[Wol] we derived the formula 
(8) 
By Yomg's formula [Y], we have for ail u  l(J, gd) that 
(9) 
 + A() -ogl«l" 
Here h(g) denot the meure-theoretic entropv of g with respect to u. 
Next we introduce topological pressure. Let Ç(J. ) denote the Banach space of 
all contiuuous functions fl'onl J t.o . The topological pressure of g[j, denoted by 
P = P(g[j, .), is a mapping from C(J, ) to  (sec [.] for the defiuition). The 
variational principle provides the formula 
(10) P(g,j,)= sup (h(g)+ f du. 
If a mesure u¢  I(J,g[j) achieves the supremmn in equation (10), that is, 
() 
it is called an equilibrium mesure of the potential ç. 
The topological pressure h the following properties (sec [R]). 
i) The topological pressure is a convex fimction. 
ii) If ç is a strictly negative fuuction, then the fimction t  P(9[j,tç) is 
strictly decreasing. 
iii) The topological pressure is a real analytic filnction on the subspe 
of H61der continuous filnctions, that is, when a > 0 is fixed, then 
P(g[j,.)[C«(J.R) is a real analytic fimction. Note that C  cannot be re- 
placed by C °. 
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iv) If  > 0 and  G C(J, R), then there exists a uniquely defined equilibrium 
measure u,  M(J, 91a) of the potential . Furthermore, we have for all 
,¢  C(J, R)» 
d I P(g,,,+t¢)=fjCd,. 
(12)  t=0 
We have P(g]j,0) = btop (glJ) (see e.g. [Val) , where htop (gIJ) denotes the topo- 
logical entropy of g]j. Therefore, by property iv), the equilibrium measure of the 
potential constant zero, e0, is the unique measure of maximal entropy of g]j. The 
map g bas a unique measure of maximal entropy (see [BS 1]); moreover, this ineasure 
is supported on J. \Ve conclude that oe0 is the unique measure of maximal entropy 
of g. We now introduce potentials which are related to Lyapunov exponents. We 
define 
(13) çb =/»" J - R, p log IIDg(p)lyll 
and the unstable/stable pressure functions 
(14) P/»: R--R, t P(glJ, :Ft¢;l»)  
The Julia set J is a hyperbolic set of index 1; thus the potentials :F¢ =/s are strictly 
negative. Therefore, property ii) of the topological pressure implies that the func- 
tions p/s are strictly decreasing. 
Since / are H61der continuous (see lB]), we may conclude from property iii) of 
the topological pressure that p/s are real analytic. Property iv) of the topological 
pressure implies that there exist uniquely defined equilibriuln measures ':te,,/»  
M(J, gIJ) of the potentials :Ft¢ /». 
We will need the following result about the relation between the unstable and 
stable pressure functions. 
Proposition 3.1 ([Wo2]). P(t)= P(t)-tloglal. 

(15) = 
The result follows from the uniqueness of the equilibrium measure of the potential 
te . [] 
We will use in the remainder of this paper the notation ut = u=t¢- This 
notation is justified by Lemlna 3.2. We also write A(t) = A(ut) and h(t) = h(g). 
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an(l consider A an(l h as real-vahw(l fmwtions of t. EquaIions (8) and (11) ilnply 
t hat 

(16) '"(t) = h(t) - t(t). 
Therefore, Proposition 3.1 implies that 
(17) es(t) = h(t) -/(A(/) - log I1). 

Proposition 3.3. A and h are real mmlytic. Furthermore, 
(18) --dA < 0. 
dt - 
If A is uot «onstaut. then ever.q zero of the derivative of A is isolated, and A is 
stri«tlg dccreasin 9. 

Pro@ l,(,t t0 _> 1) and let q5 ' be as in (13). \Ve define potentials ç = -t0u, = 
- (here we use thc notation of (12)). Therefore, applicatiolt of equations (8) and 
(12) implies that 
(19) dt (to)=-A(u,)=-A(to). 
Since P is real analytic, we obtain that A is also real analytic.  conclude from 
(16) thal h is also rem analvtic. The convexity of P" implies that 
(12 p  
0; 
dt 2 

(2o) 
(2) 

dA 
--<0. 
dt - 
Finally. if A is hot constant, then the uniquelmSS theorem for rem analytic functions, 
applied to the deriwtive of A. ilnplies that all zeros of the derivative of A are 
isolated. Therefore, A is strictly decreasing. [] 

Hausdorff dimensions of the neastlres et. \¥e lise the notation &(t) = 
dimn(ut). Equation (9) yields 
(22) A(t) =--+h(t) h(t) 
A(t) A(t)-}oglal" 
Thus, A is also a rem anah'tic fimction. Equations (16), (17) and Proposition 3.1 
imply that 
P(t) P(t) + t log la I 
(23) A(t) = 2t +- + 
A(t) A(t) - log la] 
From an elementarv calculation we obt.ain the following formula for the derivative 
of A: 
dA ,  dA( fO)dt [P(to)(A(to) -log lai) 2 + (P(to) + tolog]al)A(to) 2] 
(24) 
---tto = A(t0)2(A(t0) - log lai) 2 

Hausdorff dimension of J. The following result due to Verjovsky and Wu pro- 
vides a formula for the Hausdorff dilnension of the unstable/stable slice in terlns of 
the zeros of the pressure fimctions. 
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Theorera 3.4 ([VW]). Let p E J. Then U/s = dimH IVu/S(p)çJ does hot depend 
on p  J. Furthermore, t u/ is given by tbe unique solution of 
(25) PU/(t) =0. 
Equation (25) is called the Bowen-Ruclle fornmla. We refer to U/ as the Haus- 
dorff dimension of the mstable/stable slice. 
In [Wol] we proved the fornmla 
(26) dilnH J = tU + ts = su t ) ( h.(g) ) ( b.(g) ) 
where each of the suprcma on the right-hald side of the equation is mfiquely at- 
tained by the measures ut and ut», respectively. Hence 
(7) dim « - (t) + (t) 
A(t ) A(t ) -log ]a " 
Equation (27) and the uniquencss of the measures ut, ut» in equation (26) imply 
that, if there exists a measure of full dimension, then it already coincides with ut 
and ut». Thus, we bave the following result. 
Corollary 3.5. Assume m is a measure of full dimension for g. Then m = ut = 
ut». In particular, there exists af most one measure of full dimension. 

4. IEASURES OF FULL DIMENSION 
In this section we identify the conditions under which a measure of full dimension 
exists. \¥e start with the volume-preserving case. 
Theorem 4.1. Let g be volume-preserving. Then t  = t s, and oet is a measure of 
full dimension for g. 
Pro@ We have lai = 1. Therefore Proposition 3.1 implies that pu = p. Thus, 
Theorem 3.4 yields t u = t . Therefore, by equations (9) and (27), we conclude that 
dimH(vt) = dimH J, which implies that ut is a lneasure of full dimension. [] 
Remark. As noted in the introduction, in the volmne-preserving case, the existence 
of a measure of flfll dimension was already shown by Friedland and Ochs [FO]. 
They proved that the existence of a measure of full dimension follows from the 
fact that I detDg'(p)] = 1 holds for every periodic point p with period n. They 
also observed that in this case the measure of flfll dimension is equivalent to the 
t-dimensional Hausdorff measure, where t is the Hausdorff dimension of J. 
We now consider the volume-decreasing case. Let ,9 denote the set of ail saddle 
points of g. Note that J = S. sec [BS1]. For p  ,9 with period ri(p) we denote 
by u/,(p) the eigenvalues of Dg'(V)(p), where I*(p)l < 1 < Itu(p)l. In the next 
theorem we provide equivalent conditions for the existence of a measure of full 
dimension. 
Theorem 4.2. Assume g is volume-decreasing. Then the following are equivalent. 
i) g admits a measure of full dimension. 
ii) The unstable pressure function pu is affine. 
iii) The stable pressure function P* is affine. 
iv) The measure of maximal entropy is a measure of full dimension for g. 
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v) The quantity 

(28) 

is independent of the periodic point p E $. 

Pro@ il)  iii) follows from Proposition 3.1. 
i) :=> il) Let us assume that g admits a measm'e of full dimension. It is shown 
in [Wo2] that t » < P'. Corollary 3.5 ilnplies that utu = ut» is the measure of full 
dimension. So A(U) = A(tÆ), and this in turn implies by Proposition a.3 that the 
function A(t) is constant. Therefore, equatJon (19) inlplies that P' is affine. 
il) + iii) => iv) Recall that u0 is the torique memsure of maximal entropy for g, 
sec Section 3. The topological entropy of gla is equal to logd (sec [BS3]). Thus 
P"(0) = Ps(0) = logd. Equation 19 and Proposition 3.1 imply 

(29) dP 
dt (0) = -A(O), 
(ao) dt" 
dt (0) = -A(O) + log I,1. 

Since pu and P" are aflïlm, Theorem 3.4 and equation (26) ilnply 

logd 
logd 
(31) dimH J 
- 
A(O) A(O) -logla I " 

But by Young's formula (9), the right-hand side of (31) is equal to dimH(u0). Thus, 
u0 is a measure of full dimension. 
iv) =:> v) If u0 is a measure of full dimension for g, then, bv Corollarv 3.5, we 
bave u0 = vt. Moreover, it follows from Theorem 3.4 that U > 0. Therefore v) 
follows from Proposition 4.5 of [BI. 
v) =:> ii) follows again from Proposition 4.5 of [BI. 
Finally, iv) =:> i) is trivial. [] 

Rernark. As mentioned in the introduction, it is very likely that for volume-decreas- 
ing maps a measure of fldl dimension never exists. This can be seen by using 
property v) of Theorem 4.2, because if such a map g exists, then the parameter 
defining g provides sohltions of a countable infinite set of algebraic equations, which 
is indeed a very strong conchlsion. 

Let us a.ssume that a volume-decrea.sing nmp g admits a measure of full dimen- 
sion. Then it follows by equation (28) that 9 belongs to a real codimension one 
algebraic subset of parameter space. Using a perturbation argument, it is not too 
hard to see that we tan find arbitrarily close to 9 a nmp 9' for which (28) does not 
hold: in particular, 9' has no measure of fifll dinension. Here we mean close with 
respect to the topology on hyperbolic parameter space induced by the parameter 
of the mapping (sec [Wol] for details). On the other hand, if g admits no measure 
of fidl dimension, then (28) does not hold for 9- It can be easily shown that there 
is a neighborhood of 9 such that for each map 9' in this neighborhood (28) does 
not hold. Therefore, there exists a dense open subset of parameters admitting no 
measure of full dimension. We leave the details to the reader. 
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5. IklEASURES OF lklAXIMAL DIMENSION 
In this section we establish the existence of a measure of maximal dimension. 
We first consider the volmne-preserving case. In this situation it follows front 
Corollary 3.5 and Theorem 4.1 that ut is the unique nmasure of futl dimension 
for g. By definition, every measure of full dinwnsion is also a measure of maximal 
diinension. Therefore, we obtain the foltowing. 
Theorem 5.1. Assume g is volume-preserving. Then ut,, is the unique measure of 
mazimal dimension for g. 
We now consider the volume-decreasing case. The following theorem is the nmin 
result of this paper. 
Theorem 5.2. 4ssume g is volume-decreasing. Then there ezists a measure of 
mazimal dimension for g. If rn is a measure of mazimal dimension for g, then 
there exists t s < t < t ' such that m is the equilibrium measure of the potential 
-tdp , that is. m = 
Pro@ Since g is volume-decreasing, we bave t  < t ' (see [Wo21). 
We first assume that g admits a measure of futl dimension. In this case apptica- 
tion of Theorem 4.2 and Proposition 4.5 of [BI implies that ut does hOt depend on t. 
Moreovcr. ut is the unique measure of full and therefore also of maximal dimension. 
Thus, the theorem holds. 
We now assume that f has no measure of full dimension. Thus, by Theorem 4.2 
the flmctions pu/s are not affine. [] 
Assertion 1. There exists e > 0 such that A is strictl increasing on [0. t  + e) 
and strictly decreasin9 on (t  - e, oc). 
Proof of Assertion 1. Theorem 3.4 and the fact that P/* are strictly decreasing 
functions [property ii) of the topological pressure] imply that P(t) > 0 for all 
t 6 [0, t). Analogously we bave P(t) > 0 for all t 6 [0, U). We conctude from 
Proposition 3.3, equation (24) and an etementary contimfity argument that there 
exists e > 0 such that 
(32) --dA > 0 
dt - 
in [0, t  +), and alt zeros of the derivative of A in [0, t  +) are isolated. Therefore, 
A is strictly increasing in [0, t  + ). A sinfilm" argument shows that there exists 
 > 0 such that A is strictly decreasing in (t  - , oc). 
Assertion 1 implies that there exists t*  [t  + , t  - ] such that 
(33) dimH(ut. ) = sup A(t). 
t>0 

Assertion 2. The measure et* is a rneasure of maximal dimension. 
Proof of Assertion 2. Let (m)eN be a sequence in M(J, g[j) such that 
(34) lira dim.(mk) = 5(g). 
By Assertion 1, we may assume, without loss of generality, that dimH(0) = A(0) < 
dimH(m) for ail k  N. Recalt that oe0 is the unique measure of maxinmt entropy 
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of g (sec Section 3). We now may conclude by %ung's formula (9) that 
(35) A('0) > A(ruk) 
for ail k E N. Again by Assertion 1, we may assume, without loss of generality, 
that dim(,) = (t ) < dim(m) for ail k  N. Equation (26) implies that 
for MI k E . OEherefore, mg's formula (9) hnç]ies tht 
for MI k E . Equations (36) md (3ï) hup]y tiret h,(g) > h(t"), md therefore 
gh by equatio (30) we obth 
for ail k  N. Since A is COlltinllOllS, cquations (35) and (38) imply that for ail 
k  N there exists t  (0, t") such that 
(39) (,,,) = (t). 
Thus, the variatiolml principle (10) implies that 
h (g)  

(40) 
hence 
(4) 

dimH(mk) < A(tk) 
for all k E N. This implies that 
(42) dilnH(m) _< dilnH(,t-) 
for all k  N. We conchlde that oet- is a measure of maximal dimension. 
To complete the proof of the theorem we bave to show the following. 
Assertion 3. For every measure m of maximal dimension there exists t s < t < t u 
such that ru is the equilibrium measure of the potential -top u. 
Proof of Assertion 3. Let mbe a lneasure of nmximal dimension. We apply to m 
(instead of mk) the sanie argument as iii the proof of Assertion 2. This implies 
that there exists t  (O,t ) Sllch that A(m) = A(t). Since dimH(ru) >_ A(t), we 
may deduce bv equation (9) that hm(g) >_ h(t). On the other hand, since u« is the 
equilibrium measure of the potential -top u, we may conclude by (10) and (11) that 
h,,(g) <_ h(t). Hence hm(g) = h(t). Therefore, the uniqueness of t.he equilibrium 
measure of the potential -te u implies that ru = ut. Finally, Assertion 1 implies 
that t  (t , tu). This completes the proof. [] 
Remarks. The following heuristic argument was given by McCluskey and Manning 
[MM] to state the existence of a measure of maximal dimension in the case of 0 2 
axiom A diffeomorphisms of real surfaces. Since the entropy map is upper senti- 
continuous, it tan be shown that the map , -, dimu(,), defined on the set of all 
ergodic invariant measures, is also upper semi-continuous. It is now suggested in 
[MM] that this implies the existence of a measure of maximal dimension. To nake 
this argument rigorous we need to show that there exists a sequence of ergodic 
invariant measures m with dimH(m) -- (g) having an ergodic weak* limit. 
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Whether this holds is not clear siuce the set of ail ergodic iuvariaut measures is 
dense lu the set of all invariant measures with respect, to the weak* topology; see 
Proposition 21.9 in [DGS]. In particular, the set of all ergodic invariant measures 
is hot close& 
In the following we describc properties of the measures of maximal dimension. 
Corollary 5.3. Assume that g admits no measure of full dimension. Let t  0 be 
such that t is a easure of maximal dinension. Then 
P(t) loga 
(43) 5(g) = 2t + 
Pmof. By Proposition 3.3, equatiou (24) and Theorem 5.2 a necessary condition 
for oe being a nmasure of nm.xima.1 dimension is 
(44) P'(t)(A(t)- log ]al)  + (P(t) + t log]a])A(t)  = O. 
Therefore, the result follows from equat.ion (23). 
Corollary 5.4. The set of ail measur«s of maximal dimension is finite. 
Proof. Assume first that g admits a mcasure of flfil dimeusion. Then, this measure 
is the unique lneurc of maximal dimeusiou, If g has no nmure of flfll dilnension, 
then the function A is a non-constant real analvtic function on [0 ). Thcrefore, 
it follows fron the uniqueness theorem for rem analytic functions that A has only 
finitely many maxima in [t , t]. The result follows from Theorem 5.2. 
Corollary 5.5. Every measure oe of mazimal dimension is Bernoulli. 
Proof. Since g[j is topological mixing (see [BSI]) the result follows from the fact 
that  is an equilibrium measure of a HSlder contiuuous potential (see [BI, Thm. 
4.1). 

6. DEPENDENCE ON PARAMETERS 
Let A denote an open subset of C k a.nd let (gX)hEA be a hololuorphic family 
of hyperbolic polynonfial automorphisms of C 2 of fixed dynamical degree d > 1. 
We denote by Jh the Julia set, by ah the Jacobiau determinant, and by pï/S the 
unstable/stable pressure functions of gh. We also write Ah (t) instead of A(t). First, 
we show that 5(g) depends continuously on the parameter of the nmpping. 
Theorem 6.1. The function A H 5(gh) is continuons in A. 
Proof. Let Ao  A. The result of [VW] implies that there exist e > 0 and a real 
analytic function 
(45) ç: B(A0, e) x [0, oe) --, R, 
such that P(A,.) = Pï for all A  B(A0, e). Therefore, equations (19) and (23) 
inlply that 
(46) P : B(A0, e)× [0, x)-- R, (A,t) H Ah(t) 
is also a real analytic function. Now we may conclude by Theorem 5.1 and Theorem 
5.2 that 
(47) 5(gh) = max /P(A,t). 
tÇ[o,2] 
The result follows by au elementary continuity argument. [] 
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Renmrk. McCluskey and Manning [MM] considered C 2 Axiom A diffeomorphisms 
of rem surfaces. Thcy showed that for these mappings 6(g) depends continuously 
on the nmpping with respect to the C 2 topology. 
Finally, we show that (g) depends plurisubharmonically on the parameter of 
the mapping. 
Theorem 6.2. The function A  (g) is plurisubharmonic in A. 
Pro@ Let go E A and let L be a complex line in C k containing go. Then there 
exists a holomorphic family (g)eD, where D is a disk with center 0 in C such that 
{g: : A E D} is a neighborhood of go in L (3 A. If the radius of D is small enough, 
then there exists a family (n)eD, where each n is the uniquely defined conjugacy 
between g0[& and g:[&. Therefore, T: = (n:). defines a family of bijections 
from M(,lo,go[do) to M(J,g[&). hloreover, we bave h(go) = bT()(g) for ail 
u e ll(,lo,go[&) and all A  D (ste [Wol] for thc details). In [Wol] we showed 
that if u  M(Jo,go[&) is fixed, thon A  A(T:(u)) is a harnlonic flmction in D. 
Wc concludc by Yomg's formula (9) that 
(48) () = s,lp (h() h() ) 
The fimctions ) - A(T:(,)), ) - A(T:(,)) - log la:[ are harmonic in D. Note 
that x - x - is a convex fimction on Il'. This ilnplies that t|le functions 
A(T(oe)) -, ,X  (A(Tx(oe))- loglaxl) - are subharlnonic in D. The continuous 
function & - 6(g:) is therefore given by the supremum over a fanfily of subharrnonic 
functions. We conclude that the flmction )  (9:) is subharmonic in D. This 
completes the proof. [] 
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HAUSDORFF DIMENSION AND ASYMPTOTIC CYCLES 

MARK POLLICOTT 

ABSTRACT. We carry out a multifractal analysis for the asymptotic cycles for 
compact Riemann surfaces of genus g _> 2. This describes the set of unit 
tangent vectors for which the associated orbit has a given asymptotic cycle in 
homology. 

0. INTRODUCTION 

For many years Hausdorff dilnension ht4s played an important role in understand- 
ing some subtle features in both geolnetry and dynalnical systelns. In this note we 
use it as a tool in studying geodesic flows and the homology of surfaces. Let V be 
a compact oriented Riemalm surface with genus g >_ 2 and consider the geodesic 
flow çbt : SI V -- S1V on the unit tangent bundle. We can associate to each 
invariant probability measure/t a Schwartzlnal asylnptotic cycle A, 
By Poincaré duality we can identify H(V, IR) = H(V, IR) * and we tan trivially 
identify H1 (V. IR) = IR 2g. 
There is a particularly simple geolnetric wa.v to understand A, (cf. [Su]). Given 
v  SIII and large T we can close up the orbit segment b[0.r]v by an arc of 
uniformly bounded length to get a closed curve 7,,T, sav. Then for allnost every v 
(with respect to/z) we have that [7,r]/T --' A, as T --, oe. The range of values 
/3 C H(V, IR) that can occur for different ilwariant measures is the "unit ball in 
the stable norm". We shall prove the following result. 

Theorem 1. For   int(B) in the interior of l. the set of unit tangent vec- 
tors X_ C S V for which the limit exists and equals c is dense and uncountable. 
Moreover, the map int(B)   --, dim(X_) is analytic. 

Our approach uses a multivariable multifractal result on interval lnaps (Lemma 
5), for which we give a simple proof based on the elegant argulnent in [PW]. Pre- 
viously, Pesin and Sadovskaya studied multifractal analysis for Anosov flows [PS], 
but the exact type of multivariable results we require are hOt covered there. On 
the other hand, Barreira, Saussol and Schlneling have developed a general theory 
of multivariable multifractal analysis [BSS], but their results are formulated from 
the point of view of entropies, rather than Hausdorff dimension (cf. [Je2]). 
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1. ASYMPTOTIC CYCLES AND THE SET  
We l)egin with some general I)ackground and usefid results on asymptotic cycles 
The following defiuition was introduced bv Schwartzman in 1957 [Sc]. 
Defiition. To ea«h b-invariant measure /z we associate a linear functional A u  
H(t;) * (= q(V,)) by 

a,,() = f,()d(), 

where co is a closed 1-form repvesenting an element [coi  H 1 ( ) in the de Rham 
«ohomology. kb call A u the asymptotic cycle for 
The following result is well known. 
Lemma 1 [Sc]. If A is the Liouville measure, then Ax 
Proof. Since the proof is simple, we include it for complcteuess. Geodesic flows have 
a natural involution i : SI'  qlV so that the image i(v) of a unit tangent vector 
v h the saine bse point, but points in the opposite direction. Since i,A = A, it 
is ey to see that A() = f(v)dA(v) = 0 for ail closed 1-forms 
For other simple examples of Anosov Hamiltonian flows the Liouville mesure 
will still be preserved, but the corresponding asymptotic cycle may well be nonzero. 
D«finition. The Federer stable norm Il" Il on H,(;) is defined bv 
Let B C HI(g,N) denote the closed unit ball with respect to the stable notre. 
A well-known classical conjecture (related to the Hopf conjecture cf. [BI]) states 
that the stable norln for a torus is Euclidean if and only if the metric is fiat. Bangert 
showed that the stable norln Of a 2-torus is diffel'entiable in a rationM direction if 
and only if the corresponding nfilfilnizing geodesics foliate the torus [Bai. The 
set B w st.udied for surfaces of higher gems bv McShane and ivin [M] and 
Msart [Mail, [Ma2]. M«Shane and ivin showed that thestable norm for a once- 
punctured torus is never differentiable in a l'ational direction, and Mssart extended 
this to compact oriented ielnalm surfaces l" with genus 9  2. 
Lemma 2 [Ma2]. The se B is closed and convex. However. B is no stricl convex 
and he boundar9 is no differe¢iable in ra¢ioal directions. 
Let us consider a more dynalnical viewpoint. For each (directed) closed geodesic 
 we can sociate its real homolo class [] e HI(çN) and its lenh l(). Two 
equivalent, presentations of B C Hl(lçR) are (cf. [Mal]) 
B = {A. :  = ç - invariallt probability} 
= l() "  = is a closed geodesic . 
Observe that by fixing a suitable bis of (harmonie) 1-forms ,..., 
we tan write A, = 11 + ,.- + a2929 and therefore represent A, in coordinates 
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as a_q_ = (c1 .... , a2g). We can now define flulctions fi " SII" - I (i = 1 .... ,2g) on 
the unit tangent bundle by ri(v) = wi(v) and rewrite 
-- T--oo  fj(¢t'v)dt = %, for j = 1 .... ,29  
We briefly recall the definition of Hausdorff dimension. Let (}; d) be a metric 
space. Given e, 6 > 0 we can defille 
H(}')=inf{ Zdianl(U)sUeu "L/=6-coverforY}, 
where the smnmation is over all covers L/consisting of open sers of diameter at most 
6. \Ve ca.n then let Hs(Y) = lim6-0 H(Y) and define the Hausdorff dimension of 
}" by dim(Y) = inf {6 > 0" H'(Y) = 0}. We tan write S11" as a disjoint union 
e H (V,X)- {0 } 

where 
(a) 
(b) 

Xo = {v e X  limT--,+oo['Tv,T]/T = O} ]las fllll Liouville nleasure; 
Xa_ = {v  X" limT--,+c[')'v,T]/T = g_} has zero Liouville lneasure for 
_a¢O: 
(c) X is tlle set of points for which [7,T]/T does hot converge. 
We recall that a ç-invariant probability measure p is a Gibbs measure (for a 
H61der continuous function F " S1V  , say) if 
for all ç-invariant probability measures . The following more precise result sub- 
sumes Theorem 1. 

Proposition 1. 
(1) The Hausdorff dimension dimH(Xa) satisfies dilnH(X_) _< 3, with equality 
if and only if a_q_ = O; 
(2) The Hausdorff dimension dimH(X_) depends real analyt, cally on a_q_  
int(/3); 
(3) For a_a_  int(B), the set X a C Si I" is dense and uncountable and carries a 
Gibbs measure. 
Remark. By contrast, if a_ is a point iii the boundary of/3, then Aa_ may consist of 
a single orbit (corresponding to a simple closed geodesic) [Mai], [Ma2]. 
To prove Proposition 1 it is convenient to reduce the problem to one for interval 
maps. In particular, the following standard result helps to relate the flow to a 
simple one-dimensional system. 
Lemma 3. There exists an expanding C  Markov interval map T  I -- I and 
locally constant functions ¢  I --  (i = 1,..., 2g) such that: 
(1) the prime closed ¢-orbits  correspond fo prime periodic T " I -- I orbits 
Tnx = x (with at most a finite number of exceptions); 
(2) the least period of 7 is given bg I(7 ) = log[(T)'(x)l; 
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(3) tle homology class of  is given by I-ïl = (¢ï(x) ..... ¢g(x)), where we 
wr#e Cï(x) = g,(x) + ¢i(Tx) +... + ¢i(Tn-lx), for 1 < i < 2g and n  1. 
Proof. The «onstruçtion is fairly standard: so we shall only give an outline. We 
begin by «hoosing (two-dimcnsional) Markov Poincaré sections T1 ..... T to the 
geodesic flow çt " $11"  S I" [Ra], [Bwl]. Moreover, we can sume t.hat the 
se«tions are foliated 1)v stable manifolds. In particular, we tan identifv the sections 
along the stable nmnifolds so that each section Ti quotients to an interval Ii, say. 
Bv virtue of the Markov property of the sections, the Poincaré (first return) map 
on i=1  Ti indu«es a Markov interval map T  I  I, where I = i=ln il" Since 
the foliation of S1 I" l)y stable mmfifolds is analytic, we can deduce that T  I + I 
is C . There is a one-to-one «orrespondence between closed Ç-orbits ?, periodic 
orbits for the Poincard map and T-periodi« points, ex«ept possibly for the (at most) 
finite mmli)er of clos('d $-orbits that pass through the i)oundary of a section. 
For l)art(2), it is an esy ol)scrvation that since the surface has constant curvature 
 = -1, say, the h,ngth /() is equal to the expansion «oecient (or Lyapunov 
exp(ment) in the mist.ai)le directi()n around the orbit. If  corresponds to the 
perio(lic orbit Tnx = J-, then the associated expansion coecient around the 
ori,it is l)recisely log I(T)'(x)]- 
For part (3), we fi)llow a construction of anks [Fr].  can choose a fixed 
reference point x0  I" and contitmous paths Pi " [0, 1]  V (i = 1 .... , k) such that 
pi(O) = Xo and pi(l) is contailmd in the image (Ti) under the canonical projection 
g " SIV  V. Assmning that T-(int(li))  int(l) # 0, we can sociate a closed 
curve cij consisting of the composition of p; a curve approxinmting the geodesic 
arc from u(Ti) to (T); and pl. We lct [cij]  HI(V,N) = N2a be the sociated 
element in homolo. We can then define i " I  N by the coordinate functions, 
i.e., (l(x),...,2a(x)) = [cij] when   int(Ii) and Tx  int(I2). In particular. 
we see that if Tx  Iii, for j = 0,...,n - 1, thcn 
= (ï(),...,()). 
Moreover, the functions i are constant on T-l(int(Ij)) 

To prove Proposition 1, we shall want to apply the following general result on 
Markov interval maps. 

Lemma 4. Let T  I - I be an expanding C 1÷[3 Markov interval map. for some 
0 < I <_ 1, and let ¢  I --  (lori = 1 .... ,N) be C  functions. Let a_ = 

(al,...,aN)  I N. Let 
Y_(T,I) = w  I : 

lim ¢2(w) } 
,-,+ logl(T,),(w) [ =aj, for j = 1 ..... N . 

Let B C ]R N be the range of \  log If Idm' " ' f log IT'ldm , where m ranges over ail 
T-invariant probability measures. For   int(B) we bave that: 
(1) The Hausdorff dimension dimH(IÇ(T,I)) satisfies dimH(I(T.I)) <_ 1. 
with equality if and only if  = O: 
(2) The Hausdorff dimension dimH(Ya(T, I) ) depends real analytically on c on 
the interior of B; 
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(3) For a_ E int(/3), the set Y(T, I) C I is dense, uncountable and carmes a 
Gibbs measure. 

Lenmm 4 can be deduced as a special case of Theorenls 8 and 13 in [BSS] 
(and observation 2 affer Theorem 8). However, we can present a self-contained 
independent proof of this lemma in this paper. 
Lemma 5. lire can identify dimn(X) = dimn0(T , I)) + 2. 
Proof. It is inmlediate from the definitions t.hat Xa is ¢-invariant. In part.icular, 
we ca wl'ite that «inlH(-¥) dilnH(X  ( k ) 
= i= T, ) + 1 = max<i< dinlH(X 
T)+I. hloreover, ifv  XTi, for sonm 1  i  k, then all points lieon the 
saine piece of stable inanifold in Ti as v. Since t.he foliation of each section T, is 
Lipschitz (even C), we deduce that dimH(X  T) = dimH(Y(T, I)) + 1. This 
completes the proof.  
Proposition 1 now follows frOlll Lemmas 4 and 5. A slight nlodification of this 
analysis gives the following stronger result. 
Proposition 2. Given ,  H(i',), let X, C Sv be lbe set of unit tangent 
vectors v sucb tbat limT[%,T]/T =  and limT[%,_T]/T =  (by wbich we 
denote tBe asymptofic cycle flowing backwards in rime). 
(1) The Hausdorff dimension dimH(X,) satisfies dimH(X,)  3. witb 
equality if and only if  =  = O: 
(2) The Hausdorff dimension i11 (X,) depends real analytically on ,   
int(B); 
(3) For ,   int (B), the set X, C S  is dense and uncountable and carries 
a Gibbs measure. 

Pro@ If we had chosen in the proof of Lennna 3 to change the direction of the 
flow, i.e., replace the flow et by the flow C-t, then this would bave had the effect 
of reversing the direction of the Poincaré nmp and interchanging the stable and 
unstable mmlifolds. Following the Saille steps in the construction as before we 
would arrive at another C  expanding Markov nlap S " J ---* J, say. By a similar 
reasoning to that above we can deduce that X_, ç Ti is locally diffeomorphic to 
I_(T, I) x }(S, J), for each 1 _< i G k. Moreover, we can write dimH(AÇ,_) = 
dimH(l(T, I)) + dimH(Y(S, J)) + 1. The result then follows fronl Lelmna 4. [] 

2. SUBSHIFTS AND GIBBS MEASURES 

We now need to do sonle preliminary work in order to prove Lennna 4. The 
expanding Markov interval nlap T : I ---. I can be inodelled by a one-sided subshift 
of finite type. More precisely, we define the k x k matrix A by A(i,j) = 1 if 
T-int(Tj) ç int(Ti) ¢ , and 0 otherwise. We can defiue 
E = x E {1,...,k}  A(x,,x,+l) = 1, n_> 0 
n=0 

and a metric d(x, y) = -2=0 2-( 1 -5(x, y)). We define a local homeomorphism 
a" E- --, E by (ax) = x+. The H61der map r" Eï --, I defined by r(x) = 
(X)  
Ç=o T-I is a semi-conjugacy i.e., rca = Tr. We define HSlder functions ¢ 
E - IR, for j = 1 ..... 2g by ¢ = ¢- o r. 
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To proceed we need to impose the following hypothesis on the functions. 

Hypothesis. There are 2,0 solutions bi .... , b29 E I and  E C(E, I) to 

29 
log Ir'l o  +  b/) +  o «- , = 
0. 

It is easy to check that this hypothesis holds iii out application by consider- 
ing closed orbits. In particular, the above identity would implv that the lengths 
of closed geodesics lie in the semi-group bZ + + ... + b29Z +, giving an obvious 
contradictiou. 
We define the pressure map P  C'(EA +, IR) -- IR by 
P(g)=limsupll°g( .__ exp(gn(x))),_+ -n «  
on HSlder continuous flmctions. This lnap is known to be real analytic. Further- 
lnore, lhe hypothesis allows us to deduce that the lnap t - P(-t log IT'I o - + 
q +--- + q2929) is strictly convex III,ll. We cal, now define the following. 

Definition. Given q = (q ..... q29) e II 29, we let, t = t(q) e I be the (unique) 
solution to 
P (-t log IT'I o  + qllïl ÷...÷ q2929) -= o. 

Remark. Since log IT'I > 0 we see that for fixed q the map t  P(-t log IT'I o r + 
q2 +.-- + q29¢9) is a bijection on lt. 

The aualyticity of the map P " Cr(EA +)  I and the implicit function theory 
(using the est.imate on the derivative in the next lemma) allow us to deduce that 
t(q) has a rem analvtic dependence on q. 
We also need to consider the associated Gibbs measure. Let us define a cylinder 
set in  by [x0,...,x,_] = {y Ç  " xi = y, for 0 _< i _< n- 1}. We have the 
following useful characterization. 

Lemnla 6 (cf. [Bw2]). There exzsts C > 0 and a r-im,ariant probabzlty measure 

t = pt,q (called a Gibbs measure) such that 

1 /.t[x0,... x,_] 
(2.1) C < '^ <C, VxeF. Vn> 1 
-I(T),(rx)l-(_q)e,,ï()+...+,L() - _ , 

where we denote ¢'(x) = C"/(x) + (rx) +... + i(r'-lx), for 1 < i < 2g. 

The next lennna shows that given _ E int(B), we can associate a Gibbs measure 
/.t. 

Lemma 7. 

(2.2) 

Given c  int(B), we can find p such that 
f ¢d# for j = 1 ..... 29. 
% := f og IT'I o rd' 
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Pro@ Given __a ¢ int,(B), we tan define 

{ h(m) . 
h(_a) :=sup flog,T'(x)ldm 

fo = TAnvariant probability with 

f çdm } 
flog[T'(x)ldm =  ' 

Since e is arbitrary, we deduce that 

(2.3) 

29 
P(-h(a_q_) log IT', + Z qi(-i - ai log IT'I)) _> 0, 
Considering (2.3) with q/[Iq]]2  IR2 replaciug q we can deduce 

(2.4) 

P -h(_a)log IT'I + E qi(-i -- Oi log IT'I) 
i=1 
29 
=h(#q)-h(a)Jlog[T'ld#q+ZqiJ(-i- 
 i=1  
<0 

ai log ]T'l)d#q < O. 
,; 
=0 

for ail q E IR 2. 
from the variatiolml principle that there exists an invariant probabilty measure m. 
say, with 
In particular, the function in (2.3) tends to infinity &s IIq_lloe   and, by convexity 
of pressure, we can see that there is a unique mininmm, attained at  = (), say. 
If q is the Gibbs mesure for 
29 
-h() log IT' I +  qi(a)( i - ai log ,T'I), 
i=1 
then for each 1 < i < 2 9, the ith partial derivative at q(a) is precisely 
(- logIT'l)d#q, 
which vanishes by assumption. Thus f d.q/f log ]T'(z)[d.q = . Moreover. 
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Comparing (2.3) and (2.4) we sec that 

( 2g 2g 

=0. 

2g 
In particulm', the resuh follows with t(q) = (h(a_) + 7,i=1 ai). 

Finally, we have the fi)ih)wing id('utities. 
Len,,,,a 8. H'e bave that: (1) oj = Ot/Oqj; a,,a (2) I,.(Î_)= 1. 

l'roof. Fk»r part (1) we know by thc fornmla fi»r the dcrivative of pressure [Ru] that 
Ot - 
«lli(t 
0qP(-t(g)loglT'l +q, + " ' + q)lq=q =- d,. 
Thc r('sult th('n fi)llows from the imldi('it fimcti«m thcorem. 
l'art (2) fi»lh)ws ri'oto the Birkhoff ('rgodic theorem: For a.c. (p) x we hae that 

li,n _1  logIVo-(«'-)l =/logIT'lorrdt, a,,d 

for j = 1 ..... 29. Using (2.2) we have that t'(') = 1. as required. 

3. POINTWISE DIMENSIONS 

In this section we show that Lemma 4 follows ri'oto properties of pointwise di- 
mension. This is a straightforward mo(lification of the approach in [P\V] for a single 
function. For convenience of notation, we denote ve = r*p on l. 

Definition. Given a point w  I, we define the upper pointwise dirnetsion and lower 
pointwise dimension to be 
-,,(w) = limsup logv(B(w,r)) and d,,(w) = limiuf logv(B(w',r)), 
,.0 log r ---,0 log r 

respect ively. 

The next lemma gives bomds on these densities in terres of quantities we defined 
in the previous section. 

Lennna 9. 
(i) For a.e.(v) w  )_ we can bound d.(w) > t(q) + qloq +... + q29029 . 
(ii) For all w 
The proof of Lemma 9 is presented in the next section. Assuming this result, we 
can now complete the proof of Proposition 1. 
Lemma 10. For 
above, we can idetify 
(3.1) dimH(}) = t() + qa +--- + q2a. 
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Pro@ It is a standard result that if there exists d such that d(w) _> d for a.e. (v) 
w  }, then dimH(Y) _> d [Pe, Theorenl 7.1]. Thus we deduce froIu Lemma 9 (i) 
that dimH(Y) >_ t(ql,.. .,q2g) + qla +... + q2ga2u. 
By a folklore theorem [Fa], if thcre exists d such tlmt (w) _< d for every w  }, 
this implies that dimH(Y) _> d. Taking d = t(q) + qa + ... + q2ua2u, by Lemma 
9 (ii) we get that dimH(}) OE t(q) + qlal ÷''" ÷ q29a29. [] 

The following lenmm helps complete the proof of Lemlna 4 (1). 

Lemma 11. The map a - dimt/(}_), c  iut(B), is analytic and strictly convex. 

Pro@ The analyticity of dimH (}Ç) follows from the analyticity of q(_), which iu 
turn follows fron its characterization (in the proof of Lemma 7), the analyticity of 
the pressure and the implicit function theorem. 
The convexity of q  t(q) is easily checked to be strictly convex by showing 
that the second derivative is positive dcfinite. More preciscly, by analogy with the 
one-dimensional case [Pe, p. 212] we cau compute 

D2I(q_)(Vl,V2) = 

D 2P(- log IT'I - (Dt(v)ç),- log IT'] - (Dt(v2)-)) 

f log [T']dp 

The expression in (3.1) is the Legendre transform of t(q) and thus, again, is 
strictly convex [Ar, p. 64]. [] 

By convexity we see that dim(}_) _< 1, with equality when _a = 0 [PW]. We 
tan associate a Gibbs measure m with m(X_) = 1. Since Gibbs measures are fiflly 
supported, we can deduce that }_(T, S) c I is uucountable and dense, aud thus 
X c SI V is also dense and uncountable, as required. 

4. PROOF OF LEMMA 9 

We give the postponed proof of Lemma 9. 

Proof of Lemma 9 (i). Fix e > 0. For each w  t_ we can choose N(w) sufiïciently 
large so that for n >_ N(w) we bave: 

(4.1) 

by Lemma 8 (2). For each l  N let Ql = {w" N(w) < l} and observe that 
Ql c Ql+ aud }_ = UI Ql (up to a set of zero measure). Choose lo sufiïcieutly 
large that pq(Ql) > 0 whenever l _> lo. 
For lo fixed, choosel > lo. Given 0 < r < 1, wecancover zr-(Ql) c E by 
cylinders C (i), i = 1 .... , N, of length n(i) based at points xi  zr-(Ql), say, and 
length n = n(x, r) such that 

(4.2) 

I(T(0)'(zr(xd)l < r < [(T()-l)'(zr(xd)[. 

(This called a Moran cover [Pe].) Moreover, we can assume, without loss of gener- 
ality, that the length of each cylinder is at least lo. Usiug successively (2.1), (4.1) 
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and (4.2) we can bound 

-(B(, r) c O) _< 

(4.3) 

for some constant C' > 0. By the Borel density theorem we have that for w in a 
full measure (i.e., essemial) set of Q there exists fo(w) > 0 such that 
(4.4) (B(x,r)) <_ 2(B(x,r)Q), whcnever r < fo(W). 
In particular, comparing (4.3) and (4.4) shows that for any I > l0 and a.e.(/_z) w E Qt 
we bave 
29 
(4.5) d(w) 
,.o log r - - 
j-=l 
Since l and e > 0 are arbitrarv we deduce that (4.5) holds for a.e.(u) w E }, as 
required. [] 

Remark. Notice that the only place where we used that we have used 2 9 limits 
(rather than the usual single limit as in [PW]) is in (4.1). This Olfly influences the 
definition N(w) and, thus, that of Q¢, but does hot change the usual proof in any 
other wav. 

ProofofLemma9(ii). Fix r > 0. Let 7r(x) = w  I and choose n = n(x,r) 
precisely so that 
(4.6) I(T)'(w)1-1 < r < I(T-)'(w)1-1 

In particular, zr([x0,..., x291) 
have by (2.1), (4.1) and (4.6) 

.u(, D)) > 

for some constant C" > 0. Thus 

.x) = lira sup 

2g 
logoeq(B(x,r)) < t(q) +  qj(aj + e). 
log r - - 
2=1 

r-O 

Since e > 0 was arbitrary, we deduce that -,(x) <_ t + y9= qjoj. [] 
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Remark. A similar problem relates to the set P = {j zdp  p = T-invariant} C C 
of complex nmnbers which occur as the first Fourier coecient of invariant measures 
for a C 2 expanding map T" Iç  Iç on the unit circle K = {z  C " z = 1}. The 
set 
ç = {] zd(z)  C" p = T-invariant probability} 
has been extensively studied by Jenkinson [ae], Bousch [Bo] and others, par- 
ticularly in the case T(z) = z 2. For a typical point z on the unit circle the 
Birkhofl averages  
En=l Tn(z) COIlverge to zero. We can denote h' = {x  
1 N T n 
X " limN+  =1 («) = }, which h zero Liouville measure for  # 0. 
The analysis in this note shows that for   int(ç) we bave that: 
(1) The Hausdorff dimension dimH(X) satisfies dimH(X)  1, with equality 
if and onlv if  = 0: 
(2) The Hausdorff dimension dimH(X) depends real analytically on   
(3) X C I is dense, uncomt.ablc and carries a Gibbs measure. 
For the particular case of the doubling map T(z) = z2, this problem bas been 
studied by Fan and Schmeling IFS] and the Birkhoff averages take the simple form 
n=l z2" 
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ABSTRACT. Uniform embeddabilily (in a Hilbert space), introduced by Gro- 
mov, is a geometric property of melric spaces. As applied to countable discrete 
groups, it has important consequences for the Novikov conjecture. Exaclness, 
introduced and studied extensively by Kirchberg and Wassermann, is a func- 
tional analytic property of locally compact groups. Recently it has become 
apparent I bat, as properties of countable discrete groups, uniform embeddabil- 
ity and exactness are closely relat.ed. }Ve further dcvelop the parallel between 
these classes by proving that the class of uniformly embeddable groups shares 
a nuInber of permanence properties wilh lhe class of exact groups. In particu- 
lar, we prove that it is closed under direct and free products (with and without 
ainalgam), inductive liinits and certain extensions. 

1. INTRODUCTION 

Gromov introduced the notion of uniform enlbeddability of metric spaces, and 
suggested that finitely generated discrete groups that are uniformly embeddable in 
a Hilbert space, when viewed as metric spaces, might satisfy the Novikov conjecture 
[12], [10]. Yu proved that this is indeed the case [20], [18]. 
Kirchberg and }Vassermann defined the notion of exactness of a locally colnpact 
group in terres of the behavior of its reduced crossed product functor. Subsequently. 
they developed the main properties of exact groups. In particular, they showed that 
in the case of a countable discrete group, exactness tan be reformulated entirely in 
terres of the reduced C*-algebra [14], that is, that exactness is a property of the 
harmonie analysis of the left regular representation of such a group. 
The starting point of this work is the startling fa.et that for countable discrete 
groups, uniforln embeddability (in a Hilbert space, a geometric property) and ex- 
actness (an analytic property) are closely relate& The first indications of the re- 
lationship between uniform embeddability and exactness are fotnld in the work of 
Guentner and Kaminker [13]; these preliminary steps were quickly expanded by 
Ozawa [16], Anantharaman-Delaroche [2], and others. 
We are concerned with unifornllv embeddable groups. Our inain results are out- 
lined in the following theorem (more precise statements follow in later sections), 
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which sumnmrizes the basic permanence properties of the class of uniformly embed- 
dable groups. Observe thnt the properties described are ail shared by the class of 
countable discrete exact groups [15]. Indeed, in each case it is possible to give a uni- 
fied aceount of the results for mfiform elnl»eddability and exactness; in saine cases 
out methods provide altel'lm.te proofs of the known results concerning exactness. 

Theorem. The class of coutable discrete groups that are uniformly embeddable in 
a Hilbert spa«e is closcd under subgrotp.s and products, direct limits, free products 
with amalgam, ad extersios by exact groups. [] 

The fact that subgroups and products of uniformly elnbeddable groups are again 
Ulfiformly cmbeddablc is eleentarv and quile well known; they are included in the 
statenmnt for completencss. 
Thc othcr properties are more difficult to establish. It is possible to construct 
a uniform elnbedding of a frce product (without ama.lgam) directlv from uniform 
emlcddings of thc factors [5]. On the other hand, the corresponding result for free 
products wilh alnalgam is «onsideralfly lnore diflïcult, in view of the fact that the 
common sulgroup of the amalgaln can introduce considerable distortion into the 
produet. ()ur proof is based on a suitable adaptation of an argmnent given by Tu 
in his work on Property A [19]; although we are hot able to verify a nulnber of 
assertions concerning the mctric defined in Section 9 in Tu's paper, we are able to 
adapt his argulnents t o t ho present context. The proof we give works equally well for 
countable exact groups (sec Proposition 6.8 and Theorem 6.9), and is unrelated to 
Dvkema's original proof t hat the class of countable exact groups is closed under free 
products wit h amalgam [9], [8]. Again, in the case without amalgam a considerably 
silnpler proof of this fact is now available [5]. 
The general problem of uniform embeddability of extensions is intriguing. Out 
proof that the class of mfiformly embeddable ga'oups is closed under extensions bv 
exact groups is inspired by the argument of Anantharaman-Delaroche and Renault 
showing that the class of comptable exact groups is closed under extensions [3]. It is 
unknown whether the class of uniformh" embeddable groups is closed under general 
extensions; even lhe case of a central extension of Z by a unifonnlv embeddable 
group renmins open. At present, the behavior with respect to extensions provides 
the best possibility of distinguishing the classes of unifonnly embeddable and exact 
groups. 
We draw two immediate corollaries. Since they are peripheral to our study we 
will hot establish notation or provide lhe relevant definitions; rather, we provide 
references. 

Corollary. The class of countable discrete groups that are uniformly embeddable 
in a Hilbert space is closed .rder the formation of HNN extensions. 

Proof. An HNN extension is built from free products with amalgam, direct limits, 
and a senti-direct product by Z, which is exact [17], [4]. (Sec [6] for related results.) 

Corollary. The fundamental group of a graph of countable discrete groups is uni- 
formly embeddable in a Hilbert space if ad only if each of the groups is uriformly 
embeddable in a Hilbert space. 
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Pro@ Each constituent group is a subgroup of the fuldamental group. Conversely, 
the fundameltal group of a graph of groups is built from fi'ee products with amal- 
gain, HNN extensions and direct limits [17], [4]. (See [1] for related relnarks.) [] 

2. BACKGROUND 
Let X and " be lnetric spaces, with metrics dx alld dr, respectively. A fllnction 
F : X -- }" is a uniform embeddin 9 if there exist non-decreasing fimctions p : 
N+  N+ such that lilntoe p(t) =  and such that 
(1) p-(dx(x,x'))  d-(F(x),F(x'))  p+(dx(x,x')), for all .r, x' e X. 
The space X is uniformly embeddable if there exists a mfiform elnbedding F of X 
into a Hilbert space . Unifonn elnbeddabilitv iii a real Hilbert space is equivMent 
to uilifollll embeddability iii a COlllplex Hilbert space; henceforth ve shall deal only 
with rem Hilbert spa«es. Obviously, if X is countable we lllay assmne that the 
Hilbert space is separable. 
A metric space X is locally fiite if fi»r every x G X and R > 0 the metric ball 
with tenter x and radius R is finite. In this case the metric is called proper. A locally 
finite metric spa«e is disclete (as a topological space in the metric topology). In the 
ce of locally finite metric spaces tllere are a lumber of eqnivalcnt formulations 
of mfiform elnbeddability [7], [13]; to these we Md the following simple extensiom 
which applies to any metric space and which will be our fnndalnental criterion for 
uniforln elnbeddability. 
Proposition 2.1. Let X be a mctric space. Then X is uniformly embeddable if 
and only if for every R > 0 and 
 : x  , ()x, .ch that I111 = 1 for ail x e N. and such that 
(il) lilns sup{(x,,,) : d(x,x')  S, x,x'  X} = O. 
These conditions may be replaced by 
(iii) sup{ll - ( ,, 
(iv) lilnsoe inf{ll  -x'll : d(x,x')  S. x,x'  X} = 2. 
respectively. 
Rcmark.  refer to (i) and (iii) collectively as the convergence condition; similarly. 
we refer to (ii) and (iv) collectively as the support condition. 
Pmof. The ilterchangeability of (i)  (iii) and (ii)  (iv) follows froln the simple 
observation that for unit vectors 
Assulne that X is unifornfiy elnbeddable and let F : X   be a Ulfiforln 
elnbedding in a real Hilbert space . Let 
zp(n) =   n  (n  n)  (n  n  n) ... 

md define Exp : 7-t -- Exp(7-[) by 

((5 ) 
Exp(Ç)=l@Ç@ çç@ ççç@.--. 
Note that (Exp(ç), Exp(ç') } = e(¢'¢' ), for all ç, ç' E 7-t. For t > 0 define 
x =  -l'(x)'' Ep(V ()). 
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It is easilv vcrified that (çx,(x,) = e -tlle(x)-e(x')ll2 C(msequently, for ail x, x'  X 
we have IIçll = 1, and 
(2) e -+(a('*'))= 
Letting t = (1 + p+(R)2) -, il is easy to verifv the conditions (iii) and (ii) above. 
Çonvcrscly, assume that X satisfies the conditions in the second pm-t of the 
statement. Therc exist a sequence of maps l " X   and a sequence of 
mmbcrs S0 = 0 < S < $2 < .... inçreasing to infinity, such that for every   1 
and cvery a', .r'  X, 
(i) I1-(*)11 = 1, 
(iii) I1,()- '»(')11  , 
('bi)ose a base point x0  X and dcfinc 
 =1  by 
1 
(x) =  ((x) - t («0)  (x) - (.r0) ---). 
It is hot hard to vcrify tlmt F is wdl (tefincd 
p_(d(.r,'))  I1( ") - (.')11  d(a'.,') + . fr l ', "  X 
= £ ,= - 1 k[s_.s., and the [s_,s. are thc charactcristic 
whcre p_ 
fimctions of thc sers [S,_, S). 
Indeed, let x, x'  X. If , is such that - 1  d(x,.r') < , we have 
1 1 
I1()-(.')11  =   
1 1 x,) 
Sinilarly, if  is such that S_  d(x, z') < S, we have 
liN(*)- F(')ll  > 1 ,- 1 
  
iSn--1 
Remark. Straightforward modifications of the above argmnent produce a uniform 
embedding F satisÇving the sharper estimate IIF(«) - F(x')II  d(a', a") + 6, for an 
mbitrarily chosen 6 > 0; simply replace the 1/ in (ii) by 6/2 . 
Two metrics d and d' on the set X are coarsely eqaivalent if for every  > 0 there 
exists an S > 0 such that the d -metric ball with center x and radius  is contained 
in the d'-metric ball with center x and radius S; and conversely. Equivalently, 
two metrics on X are coarsely equivalent if the identity nmp X  X is a uniform 
embedding. 
Proposition 2.2. Let d and d' be coarsely equivalent metrics on X. Then X is 
uniformly embeddable with respect fo d if and only if if is uniformly embeddable with 
respect fo d'. 

Pro@ If two nlaps are llniforln embeddings, so is their composition. 

Remark. Below we require only the fact that if Y --+ 7-/and X --+ " are uniform 
embeddings, then the composite X --, 7-/is a uniform embedding. In other words, 
we do not need to know that X and " are coarsely equivalent to conclude the 
uniforln embeddability of X from thal of Y. 
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Let F be a colmtablc discrete group. A length function on F is a nonnegative, 
real-valued function l satisfying, for ail a and b in F, 
(i) l(ab) <_ l(a) + l(b), 
(ii) /(a -1) = l(a), 
(iii) l(a) = 0 if and Olfly if a = 1. 
A length function I is proper if for ail C > 0 the subset 1-1 ([0, C]) C F is finite. One 
can construct an integer-vahwd proper length function on F as follows. Let S be 
a symmetric set of generators of F. Let i0 : S -- N be a proper function satisfying 
(ii) and (iii) above. Then 
l(a) = inf{/0(al) +.-.+ lo(an) : a = al ""an, ai e S} 
is a proper length flnction on F. Given a length function l, we define a metric d 
by d(a, b) = l(a-lb). A metric constructed in this way from a length flmction is 
left-invariant in the sense that d(ca, cb) = d(a, b), for ail a, b and c E F. Conversely, 
every left-invariant metric arises in this way from a length flnction. A length flnc- 
tion is proper if and only if the corresponding (left-invariant) metric has bounded 
geometry. Recall that a lnctric space X has bounded geometry if for every R > 0, 
there is a unifonn bound on the nulnber of elements in the balls of radius R in X. 
We require the following well-known proposition (compare [19, Lemma 2.1]). 
Proposition 2.3. Let F be a countable discrete group and let d and d' be metrics 
on F associated to proper length functions l and l', respectively. Then d and d' are 
coarsely equivalent. 
Pro@ Since the metrics are left-iilvariant, it suflîces to consider the containment, 
as in the definition, of balls centered at the identity elenent. By symmetry it 
suflïces to show that for every R > 0 there exists an S > 0 such that, for all a E F, 
if l(a) < R, then l'(a) < S. But, since l is proper, this is obvious. [] 

As a consequence of the previous two propositions, uniform embeddability of a 
countable discrete group F is independent of the particular proper length function 
used to define its metric. Consequently, we systematically omit reference to a spe- 
cific length function or metric in the statements of ail results and say simply F is 
uniformly embeddable to mean that F is unifornfly embeddable in a Hilbert space 
for some (equivalently all) leff-invariant proper metric(s). 
Finally, we draw two simple consequences. An action of a discrete group on a 
locally finite metric space X is proper if for every bounded subset B C X the set 
{ a  F " a- B f3 B ¢  } is finite. Equivalently, for every x  X and R >_ 0 the set 
{ a  F " a. x  Bn(x) } is finite. Observe that a free action of a discrete group on 
a locally finite metric space is proper. 

Corollary 2.4. Let F be a countable discrete group equipped with a left-invariant 
proper metric. Let X be a locally finite metric space equipped with a free isometric 
action of F. Then the inclusion F --, X as an orbit is a uniform embedding. 

Pro@ Let x0 E X. Since the action of F on X is by isometries, l(a) = dx(a.xo, xo) 
defines a length function on F. Since the action is free and X is locally finite, ! is 
a proper length function. Let d be the left-invariant metric associated to l. 
According to the previous proposition, the original metric on F is coarsely equiv- 
aient to d, which is precisely what was to be proved. [] 
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Corollary 2.5. Let X and F be as in the sIatement of the previous corollary. If 
X is aniformly emb«ddablc, lhen. so is F. 
Property A is a condition on metric spaces introduced bv Yu [20]. X will work 
with the following characterization of Property A. 
Proposition 2.6 ([19]). A discrete metqc space X with bounded geometry bas 
Property A if and only if for every R > 0 and e > 0 tbere exist an S > 0 and 
a Hilbert space ,alucd fmtction ( : X   suc tat for ail x. x   X we have 
Eq¢5valentl. for ever9 R > (I and e > 1 there exist a S > 0 and  : X  12(X) 
su«h that for all x, .r'  N we l,o,,e II]] = . O) as above, ad 
(iii) suppç 
R«mark. As in the case of unitbrln elnlwddalfility, we refer to (i)  the eonvewence 
condition and to (ii) a.nd (iii) çolle«tively  the support condition. 
It is «lear frOlll lhW[nopsilion lhat Property A is invariant mder coarse equiva- 
lence. , refer lhe readcr to [15] for an introduction to exact groups. Our interest 
in Proposition 2.6 is motivated bv the following result, inspired by [13]. 
Theorem 2.7 ([6], and also [2]). A countable discree grop F 
if F has Propertg A with re.specl to some (everg) lefl-invariant proper inertie. 
In analogy with Corollary 2.5 we have the following result, in which we do hot 
smne that X itself bas Properly A. 
Corollary 2.8. Let F be a comttable discrete 9rop. Assume that F acts freel 
ad isometricall on a locall finite metric space X satisfin 9 the convergence and 
support conditions of Proposition g. 6. Then F is ewaet. 
Pro@ Include ç C X  an orbit. The convergence and support conditions of 
Proposition 2.6 pass from X to the subspace F. Since ç h bounded geometry, 
Proposition 2.6 applies. 

3. LIMITS 
We begin to establish the closure propcrties of the class of countable discrete 
uniformlv embeddablc groups. In this section we treat direct linfits, our main 
result being the following proposition. 
Proposition 3.1. Let F be the limit of a directed system of countable discrete 
groups Gi - G2 - G3 -- ... in which tbe maps G - G+i are injective. If each 
of the groups G is uniformly embeddable, then so is P. 
Pro@ The proof is based on a method of extending Hilbert space valued functions 
from a subgroup to an ambient group. Specifically, let  : G --  be a Hilbert 
space vMued function on a subgroup G of a countable discrete group F. Choose 
and fix a fanfily of coset representatives x  X c F for F/G: having done so. we 
can express each element   F uniquely as a product x« ge, where x«  X and 
g«  G. The extension " of  is defined by 
. r- ,?(r/G)-®?(x), o=o ®o-o 
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Now let F be a, direct limit as iii the st.atemcnt of the theorem. Equip F with 
a proper length function lv and associated metric dç. Metrize each of the sub- 
groups G. as subspaces of F: thc metric and length function on G are simply the 
restriction of dç and lç. Observe that for a, b  F we have 
a-lb  G  aGn = bGn  Xa = 'b  F, 
and that, lu this case, 
(3) da.(9«, 9) = dr(g,, 9) = dr(x, ,q,, .rb 9b) = dr(a, b). 
X show that F satisfies the convergence aud sut»port conditions of Proposi- 
tion 2.1. Let e > 0 and R > 0 be givcn. ()btain n such that if a  F has/r(a)  R, 
then a  G" this is possible because the length function If on F is assumed to 
be proper. According to the criterion for mfiform endedda.bility, obtain a Hilbert 
space valued function   G   such that for all 9, b  G we have IIll =  and 
(i) if dG,(9, h)  R, then I1 - &Il < e; 
(ii) vg > 0 s > 0 suçh that if dG(9, b)  S, then [(9,h }1 < g- 
Lt g be tlm xtnsion of  to I" «telmd al,ove. Clarly, I111 = 1 fo a,n a e , 
and it remains to veri(v the conditions of Proposition 2.1. Let a, b  F. For the 
convergence condition, assmne that lr(a-b) = dr(a, b) N R. By out choice of n 
we hae a-b  G and, according to (3), dG (9,9b) = dr(a. b)  R. Therefore, 
For the support condition let g > 0, obtain S a in (il) above, and Slllllç dr(a, b)  
S. According to 
  (,,), ifaG=bG, 
(4) ( «, } = ( , }( «.,a } = 0, otherwise, 
we may assmne a-b  G. But then, according to (3) again, da,, (9,, 9b) = dr(a, b) 

and I( {7«, b }1 < g. 
Remark. The previous argument is easilv 
tha.t a. direct limit, as in the statement of 
exact [15]. Indeed, under the assumption 
instea.d of Proposition 2.1, we replace (il) 
qS > 0 such that da, (9, 
Using (4) and the surrounding discussion, 
byg. 

a.djusted to yield a. new proof of the fact 
the proposition, of exact groups is aga.in 
of exactness, employing Proposition 2.6 
by 
h) _> s  (,h } = 0. 
we conclude that this property is shared 

4. EXTENSIONS 

Let 1 --+ H --+ F --+ (7 --+ 1 be an extension of countable discrete groups. 
We study uniforln embeddability of [' under various hypotheses on H and G Out 
primary result, of which out other results are consequences, is the following theorem. 

Theorem 4.1. Let F be an extension of H bg G as above. If H is uniformly 
embeddable and G is exact, then F is uniforrnly embeddable. 

As corolla.ries we mention the following two results about semi-direct products. 
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Corollary 4.2. Let G and H be countable discrete groups. Let c  G --* Aut(H) 
be an action of G on H. If both H and G are uniformly embeddable and o(G) C 
Aut(H) is eeact, then the semi-direct product I" = H > G is uniformly embeddable. 
Proof. The semi-direçt product F is thc set of pairs (s, x) E H × G, with product 
(s,x)(t,y) = (s«z(t),xy). The assignment 
(,«) ,-, ((, ),.) - r --, ( > (c)) × c 
dt'filleS ail inj('ctive holOlnorphism. 
Bv the thcoreln the semi-direct product H > c(G) is Ulfiforlnly embeddable, as 
is (H  «(G)) x G. In i)articular, F is a su|)group of the uniformlv embeddable 
group and is therefore uniforlnly cm|)eddable. [] 
Corollary 4.3. Let F = Z n ) G be a semi-direct product. If G is uniformly 
embeddable, hen F is uniformly embeddable. 
Proof. Apply thc previous corollary, using thc fact that GLn(Z) is exact, [15]. [] 
Remark. C(,ntral cxt.elsions arc m()re difficult to analyze than Selni-direct products. 
()ur lh('or('ln al)plies to c('ntral e.xtelsions in which the quotient is exact. Gersten 
has shown that if a central cxt('nsion of Z is dcscribcd by a boundcd cocycle, then 
F is quasi-isomctric to the product Z × G [11]. Consequently, if G is uniformly 
embcddable, so is F. Bcyond these two rcsults, little is known. 
As remarkcd earlier, the property of mfiform embcddability of a countable dis- 
crete groul) is in(lependcnt of the proper length flmction. Consequently, we are free 
to choose these for our groups G, H and F in a convenient lnanner, which we do 
as follows. Let le be a proper length flmction on F. Define length functions on H 
and G according to 
(5) /H(S) = lv(s), for ail s E H, 
(6) l(x) = lnin{lv(a)'aFandà=x}, for allxG. 
where we have introduced the notation a -  for the quotient map F ---* G. It is 
easily verified that the nfinimum in the definition of la is attained, and that la is 
a proper length flmction on G; that IH is a propcr lcngth function is immediate. 
Denote the associated left-invariant mctrics by dr, dG and d, respectively. Ob- 
serve that thc inclusion H  F is an isometry, and the quotient map F  G is 
contractive. Finally, choose a set-thcoetic section a of the quotient nmp F ---* G 
with the property that 
(7) Ir(a(x)) =/(x), for all x  G, 
and define   F × G  H by 
(8) v(a,x) = a(x)-oa(h-x), for ail o  F, x E G. 
Lemma4.4. Let o. b  F, x  G. IVehave 
(9) dr(a,b) < da(x,h) +dG(x,) +dH((a,x),r](b,x)), 
(10) dH(l(a,x),(b,x)) <_ dG(x,h)+dG(x,))+dr(a,b). 
Proof. Let a, b and x e G be as in the statement. From the definition (8) of V we 
obtain 
 l(a, x)-v(b, x) = a(h-x) -1 a-b«(b-x). 
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The desired ilwqualities follow easily ffoto this equality, together with (5), (7) and 
the subadditivity of length flmctions. [] 

Proof of Theorem 4.1. We prove that F satisfies the conditions of Proposition 2.1. 
Let e > 0 and / > 0 be given. We show that there exists an f : F + 12(G, 7-t) such 
that I[f(a)[I -- 1 for ail a  F, and that f satisfies the convergence and support 
properties: 
Ci) if dr(a,b) <_ R. then I1 - (f(a),f(b) }1 < e; 
(il) Ve > 0 S > 0 such that if dr(a,b) >_ S, t|len I(f(a),f(b))1 < g. 
A(lapting ail argullmnt of Almltharalnan-Delaroche and Relmult [3], our strategy 
is to use a g satisf.ving the conditions of Proposition 2.6 to average an h satisf.ving 
those of Proposition 2.1 to produce f. Since (7 is exact, there exist, according to 
Proposition 2.6. a g : G --, 12(G) all(t all S G ) 0 S/lch that IIg(.)ll -- 1 for all x  (7 
and such that 
(ig) 11 - (g(.r),.q(.q) )[ < e/2, l)rovidcd dG(x,y) _ R; 
(iii) supI)g(x) C Bsc(x), fi)r ail x  G. 
We shall without comment view g as a fimction on G x G whcnever convenient. 
Since H is tmiformly cm|)eddable, tllere exists, according to Proposition 2.1. an 
h: H --,  such that [[h(s)l [ = 1 fol" ail s  H and such that 
(ih) Il -- (h(s),h(l) )l < /2, providcd d(s,t) <_ 2Sa + R; 
(iih) Vg > 0 =ISH > 0 such that if dH(s,t) >_ SH, then I{ h(s),h(t) )1 < g. 
Having chosen g and h. dcfilm f : F -- 12(G, 7-() by 

f(a)(x) = g(h,x)h(q(a,x)), for all a  F. x  G. 

Note that f(a)  l(G.7-t). It is elementary to verifv that IIf(a)][ = 1. for all   F. 
We veri" the remaining properties. 
For the convergence property, let a. b  F with dr(a. b) <_ R. Consider 

(11) 

11 - (f(a),f(b) )1 < { 

{1- (h(rl(a,x),h(,l(b,x)) ) } g(g,x)g(),x) I 
x 
+11- (g(a),g(b))1. 

Since the map F -- G is contractive we have da(h, )) <_ R, so that by (i) the 
second terre in (11) is bounded bv e/2. Observe that. according to (iii), the sure 
in the first term is over x  Bsc(5) fq Bsc(D). Recalling that IIg(a)ll = IIg(g)ll ; 1, 
we therefore bound the first terln by 

sup{ Il - (h(q(a,x)),h(q(b,x))}l'x  

From (10) we sec that for x  Bsa(à) fq Bsa(D) we have dH(q(a,x),Tl(b,x)) <_ 
2S + R, so that by (ih) this supremnln, a.nd consequently the first terre in (11), is 
bounded by e/2. 
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For the support property, let ¢ > 0 be given and ot)tain Su as in (iih). 
b (5 F be suçh that dr(a, b) _> 2Sc, + SH. Then, 

Let a, 

I 
<_ }-lg(b,x)g(b,.r)l l( hOl(a,x),h(n(b,c)) )1 
 ',p{ I(h('Ra,-r)), h(0(,)) )1" «  s(a) n Bs(b) }, 
whre agi.w, ,,e hc fact h, Ila(a)ll = I1(£)11 = 1 to obtain the second inequal- 
ity, and note tha.t l,y (iii) the sums are in fa('t over .r G Bs (ç)Bs (). From (9) we 
see that for such m we have du(q(a, .r), tl(b, x))  dr(a, )-a(,r. a)-a(«, )  &, 
so that by (iih) the suprenmm is indeed bounded bv g. 

5. FREE PRODUCTS 
The main result of this section is thc following theor(,m. 
T|leorem 5.1. Let -1 and B be countable discrete groups and let C be a common 
subgroup. If both A ad B are ,miformly embeddable, then the amalgamated free 
product A *c B is uniformly emb«ddable. 
Out strategy for proving ho thcorcn is o construct a locally finite metric space 
X that is uniformly embeddable, and on which F acts freely by isometries. The 
construction of X is based on the notion of a tree of metric spaces, which we now 
recall. 
A tree T consist.s of t.wo sets, a set I" of vert.ices and a set E of edges, together 
with two erdpoint maps t3 -- V associating to each edge its endpoints. Every 
t.wo vertices are connected by a unique geodesic edge path, that is, a path without 
backtracking. 
A tree of spaces ,Y (with base the tree T) consists of a fanfily of metric spaces 
{ .\', .\' } indexed bv the vertices v (5 I" and edges e (5 E of T together with maps 
r,v  .¥« -- X whenever v is an endpoint of e. The r«,v are the structural maps 
of .¥. We will assume, although this is hot strictly necessary, that the metrics on 
the vertex and edge spaces are integer-valued. 
The total space X of the tree of spaces A' is the metric space defined as follows. 
The underlying set of X is the disjoint union of the vertex spaces X,,; the metric 
on X is the metric envelope d of the partial metric '(e the appendix) defined by 
 d(x,y), if qv  I" such that x, y  X,, 
d(x,y) = [1, ife  E and z (5 .\' such that x = r«,v(z), y = r«,w(z), 
for all (x, y) in the domain 
Observe that D is mnple (see the appendix); this follows from our assumption that 
the underlying tree is connected, and that d is defined for all pairs (x, ) where x 
and  are in the saine vertex space. 
We call (x, y) an adjacency if there exist an edge e, with endpoints t, and w, and 
an elelnent z  X, such that cr,,(z) = x and a,,(z) = y. Using this terminology, 
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the partial lnetric can be described as being the given metric on each vertex space 
and 1 on adjacencics. 
Example 5.2. Consider the case in which the vertex spaces are metric graphs (or 
rather the set of vertices of a graph, equipped with the graph metric) and the edge 
spaces are singletons. The total space X is itself a metric graph. Indeed, it is 
the disjoint union of the graphs, together with additional edges coming froln the 
underlying tree. Precisely, the edges of X are, first, the edges in the individual 
and second, the edges e of the underlying tree: the endpoints of such an edge e are 
the images of the maps Xe -- Xv to the endpoints v of e in T. 
Rernark. The generality of the definition above is mandated by the fact that when 
considering amalgamated free products, we will encounter vertex spaces that are 
not metric graphs. 
Theorem 5.3. Let ' be a tree of metric spaces in which the structural maps are 
isometries. If the vertex spaces Xv are uni]ormly embeddable in a Hilbert space 
(with common distortion bound), then the total space X is uniformly embeddable 
in a Hilbert space. 
Remark. Quite explicitly, the hypothesis is that there exist maps p+ as in (1) such 
t.hat for every vertex v  V there exists F : Xv -- T/satisfving 
p_(dx(x,x')) <_ IIF,,(x) - F,,(x')ll < p+(dx(x,x')), fo," ail x, x'  X, 
In other words, the saine distortion bounds lnay be used for every vertex space. 
Remark. If there exist onlv finitely many distinct isometry types of vertex spaces 
(as will be the case for thc amalgamated free product), this simply lneans that 
every vertex space is unifornfly embeddable. 
Remark. When we prove the theorem we will use the fact that the existence of a 
common distortion bound on the uniform embeddings of t.he vertex spaces implies 
that the estinmtes in the fundamental criterion for embeddability are uniform. This 
follows from the proof of Proposition 2.1, specifically from the estimate (2). 
Assuming Theorem 5.3, we prepare for the proof Theorem 5.1 by associating a 
tree of metric spaces Xv to the amalgamated free product F = A *c B. The tree 
T is the Bass-Serre tree of F /17], [4]. Precisely, the vertex and edge sets of T are 
given by 
t = r/A  r/B, 
E = r/c, 
respectively; the endpoint maps are the quotient maps F/C -- F/A and F/C -- 
F/B. In other words, the endpoints of an edge (a C-coset) are the vertices (one 
A-coset and one B-coset) that contain it. 
We associate a metric space to each vertex and edge of :F as [ollows. Equip F 
with an integer-valued proper length function and associated metric. Let v  V 
be a vertex and assume that v  F/A. In particular, v is an A-coset in F; denote 
by X this coset itsel[, metrized as a subspace of F. Proceed similarly for vertices 
v  F/B. Let e  E be an edge. In particular, e is a C-coset in F; denote by X 
this coset itself, again metrized as a subspace of F. 



3264 MAR[US DADARLAT AND ERIIx GUENTNER 

The structural maps are (lefined as follows. Let the vertex v be an endpoint of 
the edge e. [llchlsiOll of cos,ts (subsets of F) provides the stru«tural map ae, v : 
Remark. The me,tric spa«e .Y is hot (isonwtric to) the graph nmtric space of the 
Cayley grph of 4 or B, or even of the restriction of the C, ayley graph of I" o the 
subset A or B. Simila.r renm.rks apply to the edge space X. Indeed. it s important 
thot we metrize X,, ad X as subspaces of F. and hot via their identification with 
.4 or B and C sin 9 the gin,en metrics on these groups. 
Let .Yç iw the total sirote of ,ly. The grouI» F acts on Xr by left multiplication. 
Precisely. fir x G X,. and a G I" we &,fine a-.r G X,,., using ordinary multiplication 
in I'. lv virtue of the fiwt that .r G , C ['. This action preserves adjacencies and. 
since in addition the nletric on I" is let-invm'iant, it preserves the partial metric. 
According te» l're»position 7.2 of thc allwndix, F a«ts by isometries on the total 
sI)açe 
Proposition 5.4. Let. t and B b«' comptable disçrele gro,ps and let Cbe a 
subgrou p. Let 1" = .1 *c B bc lhç amalg«mat«d fwe produçt. Le A be tbe tree of 
m«trc spa«es asso«iatcd fo 1" «md let Xr bc its total space, ll'e bave: 
{i) The strwtmal map.s of .l} wre isometries. 
(ii) EoeçV ,erl«x spa«'e «g .l- is isometric to one of A or B &,bzch are 
m«trized u,ilh the sub.çpo«« metrç via the inçlusions A. B C F). 
(iii) TSe action of I" by isometries on Xp is free. 
( i,) .Yp is Iocally fin#e. 
Proof. Ail of the assertions bul (iv) follow Dom lhe previous discussion. The con- 
dition (iv), while apparent, will be derived formally in Proposition 5.6. 
Proof of Theorem 5.1. According to the previous proI)osition, the tree of metric 
spaces . associated to the amalgamated free product F = A *c B sat.isfies the 
hypothesis of Thcorem 5.3, according fo which the total space Xç is unifonnly em- 
beddahle. Again a('cording 1o thc previous proposition. ç acts freelv by isometries 
on Xç. Hence, by Corollary 2.5, F is uniformly embeddable. 
X> bave reduced out main theorem concerning amalgamated Dee products, The- 
orem 5.1. to a theorem conccrning trecs of mclric spaccs, Theorem 5.3.  now turn 
attention to the t)roof of that thcorcm. X suitably adupt thc method employ 
by Tu in his studv of Property A for discrete mctric spaces [19]. 
For a tree of metric spaces ,Y the inclusions X  X of vertex spaces into the 
lotal space are, as a general rulc, ot isometric: the presence of "shortcuts'" in 
neighboring X. m W cause the distance in X between two vertices x. y  X,, to 
be considerably smaller than the distance between them in X,, itself. Nevertheless. 
for out Xç these inclusions are isometries, a fact that may be traced back to the 
manner in which the wrtex and edge spaces of .gp are metrized  subspaces of 
F, which circumvents any distortion that mW have otherwise been introduced bv 
the amalganmting subgroup. The following proposition h no analog in Tu's work 
[19]; nevertheless, we require it in order to comI)lete out arguments. 
Proposition 5.5. Let X = { X,X } be a tree of metric spaces in which tbe 
stru«tural maps X,  X are isometries. Let X be tbe total space of,g. Then 
( i) the inclusions X. + X are isometries, and 
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(ii) if (x, y) is an adjacency, then d(x, y) = 1. 
Further, for all x E Xv, y  Xw, 
(12) d(x,y)=dT(v,w)+inf{d(xo, xl)+d(x2,x3)+...+d(xp_l,Xp)}, 
where dT is the distance on the tree T and the infimum is taken over ail sequences 
xo, . . . ,Xp, where p = 2dT(v, w) + 1. and 
( i) x = xo, y = x, 
(il) (X2a-l,X2k) is an adjacency for k = 1 .... ,dT(v,w), 
(iii) x2k, X2k+l  "¥vk, lOT k -- 0 .... ,dT(v,w), and 
v ---- VO,..., VdT(v,w ) -= W are the vertices along the unique geodesic path in T from 
v tow. 

Pro@ Let v, w  V and let x  X and y  X,. A reduced path from x to y is a 
sequence of eleinents x = Xo, x, .... X2n, x2+ of X for which there exist vertices 
v0,...,v  V such that 
(i) vj  vj+, for j = 1 .... ,n, and 
(ii) x2, X2k+l  Xv, for k = 0, 1 .... , . 
Oboerve that by appropriatcly inserting and dcleting x's we may alter a path, 
without increasing its length, so as to obtain a reduced path (use the triangle 
inequality for the lnetrics on the individual X). Consequently, when computing 
distmces in X it suffices to consider reduced paths. 
Let x = x0 .... ,x2+ = y be arcduced path from x to y. Accordingtothe 
dcfinitions of a reduced path and of the domain  of thc partial mctric we obtain 
sequences of 
(i) vertices v = v0 .... , v = w in I', 
(ii) edges el, .... e in E, and 
(iii) elcmcnts z,..., z of the edge spaccs 
satisfying 
(i) x, x+l e X, for k = 0 ..... 
(ii) the endpoints of ek are vk- and v, for k = 1 .... ,n, and 
(iii) a,_ l(zk) = x2k- and a, (z) = X2k, for k = 1, .... n. 
(The sequences are uniquely determined by these conditions.) The given reduced 
path x = Xo,...,Xp = y in X lies over the edge path e,...,e in T. 
We show that in the definition of d it suffices to consider reduced paths lying over 
the unique geodesic edge path in T from v to w; the assertions of the proposition 
follow easily from this fact. Let x = x0 .... , x2,+ = y lie over the non-geodesic 
path e,..., e, in T. We show that by successive elimination of certain xi we obtain 
a shorter path lying over the geodesic path in T. Indeed. there exists i  1,..., n- 1 
such that ei and ei+ bave the saine endpoints, that is, such that vi- = vi+. We 
have 
 Xvi_ 1 ,  Xt, i,  Xvi+ 1 = Xl,i_ 1 . 
xi-1 xi+ xi+a 
We eliminate x_, xi, x+l, x+ from the given path and clailn that the result- 
ing path 
(i) is shorter than x0,...,x+, and 
(ii) lies over an edge path with fewer bktracks than e ..... e. 
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Of these, (ii) is ol)vious; the new I)ath lies over the edge path e,..., ei-1, ei+2,..., 
e, oi)tained by climinating ei and ei+l. Purther, (i) follows froln the fact that the 
strnctural nmt)s are isolnetl'ies. In particular, we have 
d(x2i-1, x2i+2) = d(zi, Zi+l) = d(xi, x2i+l ), 
from which follows 
(x_, .r+) _< d(x_, _) + (x_, x+) + (x+, x+) 
= ,t(.r2i-2, Xi-1) n u d(x2i, x2i+) + d(x2i+2, x2i+a). 

[] 

Proposition 5.6. Let X be the total space of a tree of metmc spaces X in which 
the stru«tural ruaps are isom«tries. Assume that each vertex space X is locally 
fl'uite and there is a uniform bound for the number of adjacencies of each point in 
X. Then X is locally finite. 

Pro@ In view of fornmla (12), a finite radius ball in \" can intersect only finitely 
manv vertcx spaces. [] 

6. PROOF OF THEOREM 5.3 
Thc nmin rcsult of this section is Proposition 6.8, which implies Theorem 5.3 
and at the smne rime rcprovcs thc exactncss of free products with amalgam of exact 
groups. 
 mav enlarge Ihc tree of metric spaces (if necessary) so that the underlying tree 
T will contain an infinite geodesic w starting at some bepoint. For every v  V 
let a(v) G I" be such that the edge Iv, a(v)] points towards w. For each v G V let 
); = «¢,(X) C X and I = «,«()o«  ); + X«() where e = [v,a(v)]. It 
follows imnediately from Proposition 5.5 that each f is isonletric. 
Using the smne proposition, and the notation just introduced, we rewrite the 
distance fornmla (12)  follows. Let x0  X and x o  X,. There exists a unique 
pair of nonnegative integers k,  such that a(v) = ae(v ') and dr(v. v') = k + , 
where again dT denotes the distance in the tree T. Bv symmetry we nmy sume 
thatk2f. If k21 andf21, then 
(13) 
d(xo, Yo) T  
iO 
d(x0, x) = k+f+inf + d(f«-,()(yk-), 
+  d(L(«)(), ,+,)+ d(,) 
j=0 
subject to the constraints that y  t,(v) and yj  )#(«). If k  1 and  = 0, 
then 
(14) 
e(xo, x;) = k + inf d(xo.yo) +  d(l,(.(),+l) + d(/_,(.(_,)«;)) . 
subject to similar constraints. 
Remark. Fornml (13) and (14) are the saine as 's formulas [19, Section 9]. 
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An n-chain is a sequence x = (ar0,x 1 .... ,Xn_l) with xk E X,k(v) such that for 
each 0 _< k _< n - 2 there exists  E tk(v ) satis'ing d(x,) < d(x, t(v)) + 1 
and f()() = x+. If xo  X,, the n-chain starts in X. Note that for any 
n 2 1, v G V and xo  X,, there exists an n-chain whose initial element is Xo. 
Lemma 6.1. Let xo  X and x  X« with d(xo, x) < R, and let k and g be 
' .. ) 
a i (a) nd (4). Thon t it i (-o,,...,), (,1,- , 
th.at 
lllaX { (ç0ik_lSUp d(xi, Ti+l)). ç0jg-l(Sllp d(.r}, X+l) 
Remark. In the proof of the lenmm, and at a number of subsequent points, we 
ruire the fact that if x  Xv and g  ,, then 
d(x, f(y)) = d(x,y) + 1. 
This follows from Theorem 5.5. Indeed, since (y, f,,(y)) is an adjacency, d(x, f,.(y)) 
 d(x, y) + 1. For the reverse inequality, let e > 0 raid obtain x   tÇ such that 
d(x, f,,(g)) + e  1 + d(x,x') + d(fv(x'),f(y)). Since fv is an isometry, we get 
d(x, x') + d(f.(x'), f,(y))  d(x, y). aml we me done. 
Pw@ If v = v  there is nothing to prove; so wc may assume that v ¢ v'. By 
symmctry we may also assulne that k  ; hence k 
Case  = 0: We need to prove that there exists a chain (x0, x,... ,xk) such that 
,,lax { (ço(i(k_ls,,p d(£,Xi+l)), d(xk.£, }  2k  . 
Since d(xo, x) < R, by (14) there is Yo  t such that k + d(xo. Yo) < R. The 
sequence x1 .... , x is constructed inductivcly. Let 
d(zo, go) and d(xo, o) < d(xo, };) + 1. and define Xl = f,(2o). Then 
d(xo, xl) = d(x0,0) + 1  d(xo,Yo) + 1 < R. 
Thus d(xo,xl) <  and d(Tl,X)  d(xo,x) + d(T0,Xl) < 2. Repeating the 
 (if k > 2), we find x2  X=(v) with 
smne argument for the pair of points Xl,X o _ 
(Xo, Xl,X2) being a chain, d(Xl,X2) < 2 and d(x2,x) < 22R. Continuing in the 
saine way, we obtain a chain (x0,x .... ,x) such that d(xi-l,Xi) < 2i-1 and 
d(xi, x) < 2iR, 1 < i < k. Since k  R, this completes the proof for  = 0. 
Case  >_ 1: Let Y0,--.,Y-I with yi  '(v) and y,...,y't_ with Yj'  Ya(v') be 
sequences such that the expression whose infinmm is taken in (13) is less than 
= (v')(Yt-1). Then we bave d(x0, z)+ 
R-k-g. Let z f-,()(y-l) and z' = ft- ' 
d(z,z') + d(x,z') < R, so that both d(xo, z) and d(x,z) are less than R. The 
proof is completed by applying the first part of the proof to the pairs xo, z and 
x, z and noting that 
d(z, zÆ) 5 d(x, z) + d(z), z) < 
Gin an n-chain x = (x0,l,...,x-) stting in 
n - 1,  = d(x, I(v)) and Ok = o V ... V , where we introduce the notation 
a V b = max{a, b}. For future notational convenience define 0-1 = 0. Note that if 
 .. X  
x  = (x,x,. , -1) is another n-chain starting in X,, then 
(15) d(xa_, x»)  « + 2, 
(6) 6 -6] = d(,X)-d(x;,X,,)  (,), 0  5 n- 1. 
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Lemma 6.2. Let x = (Xo, Xl,...,x,-l) and x' = (x'o,x 1 .... ,x,_) bc n-chams 
starting in X,,. Tben 
(17) Id(.rk,.r)--d(xo, xo)l<_2k(Ok_l VOk_l)+2k, 0_<k_<n-1. 
Pro@ Since the statement is obvious in the case k 
aud z - be as in the definition of n-chains. Observe that 
d(., - -   ' ' 
xk)  d(k, xk) + d(x k, xk) + d(xk, xk) 5 d(xk, xk) + 2(0k V Ok) + 2, 
d(2,x) > d(x,x) d(,.r) -' ' ' 
-' _ ' - -d(x,x) 
Since f() is an isometry we bave d(x+,x+) = d(k,), and the lemma 
follows from these incqualities bv induction. 
Given an n-chain x = (xo,.r .... ,_) and N > 0, we define, for 0  k  n-1, 
(18) ak = Çl ln(l+0k)) 
N + 
( « = o-'--,(+(-'-)(-)), 
where n+ = max{a, 0}. Note that co = 1 + (n - 1)(1 - ao). One checks immedi- 
ately that ao  a  ...  an-, 
2 
(20) cô+---+c_=, 0c_l, 

(21) (k >_ e N -- 1 == Ok >_ e N -- 1 == ak = 0 == ck+ .... = cn-i = O. 
The coefficients ck were introduced bv Tu [19] (actually we work with the square 
root of 's coefficients). Since the maps f are isometries, it is possible in our ce 
to dcfine ak explicitly  in (18). Both a and c should be regarded  functions 
a and  of -chains x = (Xo, X,... ,x_). ç will often write a and c; insteM 
of a and  . 
Lemma 6.3. Let x = (Xo.X ..... x_) and x' = (x,x,...,x_) be -chains 
starting in X. If  = nl0<i<n-1 lai-al, then 
() c-c  (), 
(il) Ic - cl   . and 
(iii) --1 I (ri+l)  
Z=0  I«- c;I   - 
Pro@ This is an exercise. For (i)  (ii) use the inequality I- l  2[a - bi. 

The following continuity property of the coefficients a is a minor variation of a 
formula in Tu's paper [19, fornmla 9.2]. 
Lemma 6.4. Let x = (xo, xl,...,x,-l) and x' = (x'o, xl,...,x,_) be -chains 

starting in N,, and assume d(xo, Xo) <_ n. Then 
7n. 2 
(22) ma I,, - ,;1 < -- 
0<k<n-- 1 -- iV 
Pro@ Denote A = In 7. çVe are going to show that 
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From these we ilmne(tiately obtain lat.- a.[ _< n(1 + A) ïn 2 
< concluding the 
proof. Of these inequalities, the first is straightforwm-d. [ndeed. using (16) and 
the property that the mal, t  (1 -  ln(1 + t))+ is -Lipschitz, we have 
 1 , 
lao- ;I 5 la- agi  d(*,,,«o)  . 
To prove the second inequality, denote f(t) = 4nt + 6n. According to (16) and 
Lemnla 6.2. we have 

1 
(24) lSt:- ([-I- d(d'/,-,./¢)_< f(Ok_ 1V Ok_l). 
Denote g,(t) = ln(1 + t), so that for ail t 2 0 we have 
(25) V,(t)  ,(f(t))  ¢(t) + A. 
Cse Ok V O 5 f(O_l V 0_1)" By symmetry we mav assmne that 0k_ 1 OE Ok_ 1. 
Thus 0e_ 1  0 k  f(0e_) raid 0_ 1  0-1  0  f(0n_). Using (25), we obtain 
0(0_1)  ,(0) {(f(0t.-1))  g'(0k-1)+ t. 
ç(0_l)  {'(0) g(f(0-l))  (0-1) + A. 
Now, (23) follows immediately ri'oto these inequalities, together with the property 
that for rem nmnbers a < s, t < b +  we have 

I(1 - s/N)+ - (1 - t/N)+l <_ I(1 - b/N)+ - (1 - 
Case Ok V 0' _ > f(Ok-1 V 0._1) " Bv .... Sylnlnetrv we mav assume that 0 OE 0. Then 
0k OE f(0k-1) OE Ok-1, and hence Ok = ak- Therefore. using (24), 
O  [0. -- 0.[ + O  [6k--6[.]+0  f(Ok-1 VOk_l)+O k 
Hence 0  0k 5 20.  obtain 
 (0)  ,(0)  ,(0) + in 
From this inequality and the property that the map t  (1 -t/N)+ is -Lipschitz, 
we obtain lac - a  '  , which ilnplies (23). 

Definition. Given R > 0 and  > 0 choose and fix n ¢ N such that 

1 
< , n > 2RR, 
an 3(R + 1) 

a.nd N G 1%1 such that 

(27) 6n(n + 1) 
OE < 3(R + 1)" 
Having done so, apply the fundamental criterion fol" uniform embeddability. Propo- 
sition 2.1, to choose and fix a familv ({)ex of unit vectors in a Hilbert space 
x = et"  with çx E  if x E X, and such that 
(28) 
3( + 1)" 
(29) lira sup{]{{v, «, _ 
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(See the remarks after Theorem 5.3 for comm(,nts on why this is possible.) Finally, 
for every '-chain x = (a'0, a'l .... ,a'-l) in X define the unit vector /x G 7Y¢ by 

(30) 

= -Z 
k-=O 

whcre, of course, the ck's are defilmd according to (19) and depend on the chain x. 

Lenlna 6.5. Let x (Xo,.rl,.. ,x,-l) and x' "' ' .. x' 
=  = (Jo, xl,  , -1) be n-chazns 
startin 9 in .\',. If d(xo,.r'o) <_ ,. then II,f' - ,f' Il < 3(R+l)" 

! ! 
Pro@ Let I = {k" c» ¢ O.c» ¢ 0} and J = {k" ck = O, ck ¢ 0}. If 1 <_ k E I, then 
ak-1 = (1 -  ln(1 + 0k-l))+ # 0: helice 0k-1 <_ e N- 1. Similarly, 0_ 1 <_ e N- 1. 
Hence from (17) we have 

(3) 
Consider 

t 

d(:,-,. -) < 2,(#  - 1) + 2, + d(*o..,';) < 3,,« u. 

 Ck Çx«  -- 
k=O 
k=0 
< 

Observe that the 's and {«'s are in orthogonal components of 7.\. We bound 
the third terre on the right using (20), (31) and (28)  follows: 
1 ( )1/2 1 (,n-1 t 2 
 («)2115-ç112   («D - supllS« 
kkI kkI 

- .-x - 3(R + 1)" 
where A = {(. 9') " d(9, ') _< a,,e , v, v' E \,  e  }. we bound the smn of 

the first two terres on the right, by 
1 
3 - Ick - «12 _< 
\k=O 

3(n + 1) 

lllaX 
 O<i<n--1 

1 
3(n+ 1) (Tt)   
x/ \ ,OE ] -< 3(R+1)' 

where the inequalities are from Lemma 6.3, (22) and (27), respectively. Combining 
these observations, we obtain the result. [] 

! ! 
Lemma 6.6. Let x = (Xl,X2 .... ,x,) and x' -= (32O, Xl»...,Xn_1) be n-chains 
with Xo  A% and Xl  X(). If (Xo, X) is a 2-chain and d(xo, xl) < n, then 
IIw"-,«11 < ,- 
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Pro@ Let 5:o = f71 (Xl) and set  = (5:o, Xl ..... x,-l). According to the definition 
(30) of q" we have 
 kk=l 
:= v (°'' ..... -'-  &o +  & , 
k=l 

where 

dk = Ck-l(Xl ..... xt) = V/fil "-.t_l(1 + (n -- k)(1 -- k)), 
'ïk = C("0, Xl, ---,xk) = (Ol'''k--l(1 + (" -- k -- 1)(1 -- 
with o = 1, o = 1 and k = ak(o,x,.. ,x) = rein{(1 1(+,) 
 N )+ <i<#}. 
For lçkn-1 wehaveeither 1--k=0, hencedk--dt.=0, or 
l+(n-k)(1 -a)+l+(-k-1)(1-6) 
< (1 -) < 
- - )(-) - 2- 
Applving the above expressions for qx and @ with this inequality, we estinmte 

1 
< +-(4-&) 2 
r/ k=l 
< +-  1 +- 
-- 
l 

hl n + 9"  e 
< \ - ] < 3(R--l)' 

1 

where the final inequality cornes flore the choice (26) of n. Apply Lemna 6.5 to 
the chains x' and , noting that d(xo, o) = d(xo, Xl) - 1 _< n, and use the previous 
inequality to conclude that 
, , 2e e e [] 

! ! 
Lemma 6.7. Let x = (xo, Xl, . . ., xn-1) and x' = (3o, Xl, ... , x,_,.) be n-chains in 
X. Then 
I(r/X,r/X'}] _< sup {l(u,u,}] " y, y' _ Xv, v _ I\ d(y, y') >_ d(xo, xo) -- 6rieN }. 
Pro@ Let k and g be as in (13). Considering symnletry, we assume that k _> g. 
Also, if k >_ n, then (tf', tf") = 0; so we assume that k < n. With these assunlptions 
n-k-1 
, = _1 Z ck+ic'e+i({'+ ' {«+,>" 
(32) (r/x, "r/" ) n 
i=0 
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lIaking use of (21), we conclu(h, that if 0-1 V 06_ 1 _> e N - 1, then ail the products 
ca+ce+ i in (32) are zero, and we are don(,. Thus we ssume that 0_ V 06_  
çN _ 1. Let m I)e the largest lllllllb('r 0  tt  ri -- '-- 1 with the property that 
0+_ v 0}+._  e N - l. By (17), 
d(x+i, a'e+i) > d(a , xe) - 2ne u 
for ail 0 < i < m. On the other hand. using (15), 
k-1 -1 
(.,., .i)  (.,.,.)-  d(.,, .,+)-  (, +)' 
k-I 
 (.,.,.ç)-  (, + 2)-  ( + 2) 
i=O j=O 
 d(x0, x'o) - 4e n. 
Combining these two in(,qua]ities, we ot)tain 
d(x+, ' ' 
.re+i) d(xo, x0)-6te N, ri)rail 0<i<m. 
Finally, arguing again on the basis of (21) s ai)ove, we conclude that the terres 
in (32) for i > m are zero. Tlms, apI)lying (20) and the previous inequality, we see 
that 
- ,u )1 < sup 
where fi = {(y,y') : y.y'  X,,,v  i d(y,y') OE d(xo,x)- 6ne'}. 

Proposition 6.8. Given R > 0 and e > O. let n, N, and (x)zEx be constructed 
in the definition. For each Xo E N. choose and fix an n-chain x = (xo, x .... , x_) 
and consider the correspondin 9 vector q* = q(,o,, ..... -). Then 
(33) suv{[[ * - ,l  [l " d(xo,«o) < R}  
(34) I(V*.,I*')I  sup{[(Su,Su,)l - d(v,y' )  d(xo, x)-6,wN, y,y ' 
Proof. The support condition (34) was proven in Lenmm 6.7. For the convergence 
condition (33) we show that for any x0, x 0  X with d(xo, x) < R and any two n- 
' respectively, we have llç*-q*'  < ê. Let k and 
chains x and x' starting at x0 and x 0, _ 
g be  in (13) and (14). Let x(i) and x'(j) be -chains whose initial elements are 
the points xi, xy' given by .... Lemma 6.1, 0 < i < k, 0 < j < g. Applving Lemm 6.5 
and 6.6 repeatedly (note tha.t since n > 2nR this is possible bv Lemma 6.1), with 
the convention that ail empty sums are zero, we ha.ve 
k--1 
, - ,'ll   () - ('+')11 + lv () - '(«)ll + 
t=0 j=O 
< (k+g+l)ê <e. 
- R+I - 
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Let us note that 
Theorem 6.9 ([8], 
common subgroup. 
A *c B is exact. 

Conclusion of the proofof Theorem 5.3. Given R > 0 and e > 0, let x0 
 r/(zo,E ..... E_I) be a map constructed as in Proposition 6.8. It satisfies the 
convergence aud snpport conditions of Proposition 2.1, as shown by (33) and by 
(34) in conjunction with (29). [] 
Proposition 6.8 also reproves the following fact. 
[19]). Let A and B be countable discrete groups and let C be a 
If both A and B are exact, then the amalgamated free product 

Proof. Given R > 0 and e > 0, let x0   (E0'E1 ..... En--I) be a map constructed 
as in Proposition 6.8. If the family (E)EeX satisfies the snpport condition (ii) of 
Proposition 2.6 uniformly with respect to v E V, then it follows from (34) that 
(?×,?x') = 0 whenever d(xo, Xo) _ S + 6ne N. In particular, the map x0  
Z](E0,E ..... E-I) satisfies the convergence and support conditions of Proposition 2.6, 
and A *c B is exact by Corollary 2.8. [] 

7. APPENDIX 
We collect several elementary results ou the coustruction of lnctrics required for 
our treatmeut of amalgamated free products. Let X be a set. A partial metric on 
X is an integer-valued function d - T) - + defined on a domain T) C X x X and 
satisfying, for ail x, y E X, 
(i) dx,y) = (y,x), and 
(ii) d(x, y) = 0 if and only if x = y. 
In these statements it is assumed that all relevant pairs belong to the dolnain D of 
'and that the domaiu T) is symmctric and contains the diagonal. 
We now associate a metric to a partial metric. The construction is analogous 
to that of path mctrics; intuitively the distance between two points is the length 
of the shortest path between them. A path from x to y  X is a sequence x = 
XO, Xl,... , X n = y such that for every 1 _ j _ n we have (Xi--1, Xi)  ). The length 
of a path is -= (xi-, x). The donmin T) is ample if for every x, y  T) there 
exists a path from x to y. 
Proposition 7.1. Let  be a partial metric on X, defined on an ample domain I). 
Define, for ail x, y  X. 
d(x,y) = in{ lent oI pats Irom x to  }. 
Then d is an integer-valued metric on X. [] 
The metric d defined in the proposition is the metric envelope of d. 
Example. If X is the vertex set of a connected graph and 'is the constant 1 on the 
domain of all pairs (x, y) that represent edges, then d is the path metric; d(x, y) is 
the smallest number of edges on a path from x to y. 
Remark. If (x,y)  ), then d(x,y) = (x,y) if and only if the length off every path 
from x to y is greater than or equal to d(x, y). If T) = X × X and d is a metric, 
then d = . 
Proposition 7.2. Let  be a partial metric on X and let   X -- X be a bijection 
with the property that 
(i) (x,y)  ) if and only if ((x),(y))  T), and 
-_-, 5",._,_, -'-" " .... (x, y)  
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Then o is an isometry for the metmc envelope. 
Proof. It suCices to show that 
d(o(x),o(y)) < d(x,y), for all x, y  X. 
Let e > 0 be given. Let (x0 .... ,x) be a path in X from x to y such that 
ji d(xj_i,xj) < d(x,y) + e. Then ((x0),...,(x)) is a path from ç(x) to 
ç(y), and we have 
Since e > 0 was arbitrary, we are donc.  
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VITALI COVERING THEOREM IN HILBERT SPACE 

JAROSLAV TIRER 

ABSTRA('T. It is shown that the statement of the Vitali Covering Theorem 
does not hold for a certain class of measures in a Hilbert space. This class 
contains ail infinite-dinmnsional Gaussian neasures. 

1. INTRODUCTION 
V'e start with recalling the st.atement of the classical covering theorenl due to 
G. Vitali, [9]. 
Theorem 1. Let ,4 C n be a set. .4.ssurne that for every x G A there is a 
seq,ence (B[x, r(x)])k of closed balls centred at .r and with radii r(x) such that 
limkoerk(x) = 0. Ten tere is an ai most countable family of disjoint balls. 
The balls in the original pal)er were considered with respect to the norm II.ll. 
In fact, the statement of the theorem above holds truc for balls in any equivalent 
nonn in N . 
Since the rime of Vitali there appeared manv generalizations of the statement 
in various directions. To mention at least one of them, now already clsical, we 
have to point out the version bed on the Besicovitch Covering Theorem. It 
extends the statement from Lebesgue measure to any «-finite measure on N; sec 
e.g. de Guzmgn [1]. 
Our aire is to study what happens if we replace the n-dimensional Euclidean 
space N  by an infinite-dimensional Hilbert space. The first result of this type is 
due to D. Preiss, [4]. He gave an example of a Gaussian measure on a separable 
Hilbert space for vhich the covering theorem fails to hold. 
One of the most important consequences of the Vitali Covering Theorem is the 
so-called Differentiation theorem. The original version goes back to H. Lebesgue. 
Employing the above-mentioned generalization of the covering theorem, one has the 
following form of the Differentiation theorem. Here, and also in the sequel, B[x, ri 
denotes the closed ball with tenter x and radius r > 0. 
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Differentiation Theoretn 1. Let g be a locally fintte measure on n and let 
f  Loc(p). Then 
(l) liln l /B f dp = f(x) p - a.c. 
,.o ,B[z, ri 
The negative result of D. Preiss [4] was later strengthened in [] by constructing 
a bounded function and a Gaussian measure on a Hilbert space such that (1) does 
hot hold. Moreover, in [6] the sanie author obtained a Gaussian mesure 7 together 
with the integrable function f  LI(7) such that the means of f over the balls in (1) 
tend to infinity unifornlly with respect o .r. 
On the oher hand, J. Tiger [8] has shown the validity of (1) for some class of 
Gaussian nleasures on a Hilbert st)ace and fi)r all L p funcions, 1 < p < . This 
result could illdicate lhat there is a chance for having the Vitali Covering theorem at 
least for some infinite-dilnensiol]al Gaussial nleasures. However, Theorem 1 below 
lnakes clear that it is lmt the case, and tha.t Preiss' example [4] was hot accidental 
froln this point of view. 
Before sta.ting Thcorem 1 we recall he concept of a Vitali system. 
Definition. Let A C X be a sui)set of a lnetric space X. A family 
v c {N.,d I  e A,  > 0} 
is called the Vitali 8ystenl on A if for every x  4 alld for every ge > 0 the system 
F contains a ball B[x, ri with r N e. 
Theorem 1. Let H be a separable Hdbert space and let 7 be a Gaussian measure 
with dira spt? = oe. Then for every  > 0 there exts a çtali system F on spt 
such that any disjoint subfamily 8 C ç satisfies 
7(U ) & .e., 7(spt 7U ) 1-. 
Theorem 1 is an ey consequence of the following Proposition 1, which is formu- 
lated for more general mesures than the Gaussian ones. We make some comments 
on the other consequences of Proposition 1 at the end of this section. First, hower, 
we shall introduce some notions and notation. 
The symbol sptt will denote the support of a measure . The projection Pu of 
the measure p onto a closed subspace U of the Hilbert spe H is defined by the 
fornmla 
u,4 = le,  (A), 
where u" H  U denotes the projection and A C U is anv Borel set in U. If U C 
H is a finite-dimensiolml subspace, then we shall denote by u the corresponding 
dira U- dimensional Lebesgue measure. 
} shall also mention some bic facts concerning Gaussian mesures. 
Definition. A probability mesure u on the real line  is called a Gaussian mea- 
sure, if either u is the Dirac lneasure supported at 0, or it h the Radon-Nikodm 
derivative with respect to the Lebesgue mesure of the form 
du _ 1 
d -  exp 
for some  > 0. A Bord probability 7 on a separable Hilbert space H is callM 
a Gaussian measure if every projection of 7 onto a one-dimensional subspace is a 
Gasian measure. 
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We consider the Dirac measure to be a Gaussian measure only for convenience. 
It enables us to include among the Gaussian measures also the measures that are 
supported by a proper subspace of the Hilbert space H. 
Let 7 be a Gaussian measure on H. The covariance operator S 7  H -- H is 
defined by 
Ju(x,h) (y,h) dT(h), x,y e 
H. 
The operat.or S 7 is always nolmegative ((STx , x) _> 0), selfadjoint and nuclear; sec 
e.g. [3]. If spt7 = H, the covariance operator is even positive definite. Iu that case 
the eigenvectors of S form an orthonormal basis (en) of H with Ihe following nice 
property: If 7n is the projection of 7 onto the line spanned by en, then 
(:)  = I-[n- 
Such representation of / as a countable product will be uscful. 
Definition. Let r > 0. The symbol (r) denotes the set of all disjoint families of 
closed balls in II of radius r > 0, 
(r) = { I  is a disjoint family of halls of radius r}. 
Proposition 1. Let H be a separable Hilbert space and let I be a flnite Borel 
rneasure on H with the followin9 propertg: For ever 9 n  N there is a finite- 
dimensional subspace U C H such that 
(i) dimU>_n, 
(ii) Iu is absolutely continuous with respect to the Lebesgue measure ..u on U, 
(iii) / _< #u x #u'- 
Then 
.m u,{,(U)I   ()} =o. 
-.ll, 0 
Proof of Theorem 1. Without loss of generality, we may obviously assume that 
spt7 = H. If we recall the representatiou (2) of a Gaussian measure as a countable 
product of one-dimensional Gaussian measures, then we sec that the conditions (i) 
- (iii) of Proposition 1 are satisfied. Indeed, let. (en) be the orthonormal basis of H 
consisting of the eigenvectors of the covariance operator $7. Then for an), n  N 
we put U = span{e .... , en}. The conditions (i) and (il) are obviously truc and in 
the condition (iii) we even obtain equality. 
Let  > 0 be given. By Proposition 1, there is a decreasing sequence of numbers 
r "% 0 such that 
We define the following Vitali systen: 
Let S C V be any disjoint subfmnily. Then 
8= U8' 8={Be81radius(B)=r}- 
kN 
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Now, by using (3), 

 (u)=  . (u ) <-   --.. [] 
k=l k=l 

ReTnark. Note that the finite-dimensional subspaces U C H ffOlll Proposition 1 
ueed hot be nested. Also, if we choose for any n  N the correspondiug subspace 
U with the prop«rties (i)-(iii), theu the liuear si»au of {U [ n  N} need hot be 
dense in H. llence the conclusion of Proposition 1, aud consequently non-validity 
,f the Vitali Cov«riug lh««,rem. «au he obtained e.g. for lhe following type of 
lllOaSlll'eS: l,ct 
tt = tlo  tltî 
1o an orthogolml de«OlnpOitiOli of H Sll('h that dira H0 = oe. Let (p,,) be anv 
se<tutrice «[" al»s«»hlte}y continloll l>robalfility lllOa.sllrOS O11 . XX consider the 
11 IOaslIro 

tre the space IR r. Siuce Ho  g2 C IR r, bv the 0- 1 law there are only two 
possildlities: either ptlo = 0 ¢r pHo = 1. Assmne the latter. In that case for 
mlfitrary finite measure v O11 /-/ the product p x ve on H is an exalnple of a 
nmasnre satisfying the asSlllllptions of Proposition 1. 

"2. LEMMATA 

Let U C H be a closed subspace of the Hilbert space H, and let  be a family 
of disjoint closed halls in H of radius r.   93(r). We denote by u the falnilv 

,, = {UnB I B e 3}. 

Obviously. u is a disjoint familv of closed halls in U of radii at most r. 
The first Lemma establishes oue simple geometriçal relationship among the balls 
in tr. 

Lemma 1. Let U C H be a subspace of a separable Hilbert space H and let 
 e (1). Let B[,,.r,] a.,,d B[w2.r] be two digeret halls from u. If eithcr 
2r < fo or 2r 2  ri, then 
I1, - ,211 2 ,, + , + (- 3) lliaX{rl, ,'2}. 
Pro@ Since the halls B[, r] and B[v2, r2] belong to u, there are two unit halls 
B[xl, 1] and BIg2, 1]   such that 
[,,,,,]=vnB[,.,.] ana B[u2.2]='nBlz,]. 
Also. since g = U ¢ U z, one has 

 ri = Ul + ri and x2 = a2 + v2, 
where tl. u9 Ç U and ri, t, G U -k. Bv the disjointness of the ba.lls in  it is readily 
seen that 

(4) Ilul - u=ll 2 + I1' -v211 = = IIx, -.,'=11 = _> 4. 
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Note that rî = 1 -IlVl[[ 2 and r = 1 -IIv2[[ 2. Using this in the estimate (4) we 
obtain 
I1, - 2112 _> ee ÷ ,î ÷  + I1,11  + I111  -I1 - 11  
(5) = oe + rî + r + oe(v,v=)  oe + rî + r - oell'll IIv=ll 
+  +  - ((a - .)( - ). 
2 
Without loss of generality, we nmy assume that r2  ri. Then the assumption in 
Lemma 1 implies that even 2r2  r. Let  = ( - 3). In order to prove that 
II - 1  ,,( +6) + , 
we are going to show that 
,,, - ll  - (,(1 + 6) + )  0. 
To this end, we use the estinmte (5)" 
1 - ll  - (,(1 + 6) + 
 e + î +  - e¢1 - î)¢ - ) - (,(1 + ) + ) 
= e - e(1 - î)(1 - ) - ((1 +) - a) - »,(1 +) 
= g(r, r2). 
We shall have to find the minimal value of the flmction g(r, r2) on the set {(r, r2) 
0  2r2  r  1}. Some elementary calculation reveals that the flmction r2 
g(r, r2) is nonincreasing on [0,  
r]]. One more calculation gives that the function 
r g(r,  
 r) is nondecreasing on [0, 1] provided that 
5 
(1 + 6) + (1 + 
This condition is guaranteed by out choice of . Hence the nfinimal value of g(r, r2) 
is attained at the point (0, 0) and is equal to 0. This completes the proof. 
The next lemma estimates the Lebesgue mesure of the intersection of two balls 
in a special position. The symbol (n) denotes the volume of the unit Euclidean 
ball in , 
(,,) = ,B[0. ] - r( + /)" 
Lemma 2. There is a Ao > 0 such that for any x   with ]]x][ = 3 and 
0 <   o we have the following estimate: 
( ) 
 [0,+][.(+a)l (--) 0 % (+) . 
Pro@ Let z = (a. 0 ..... O)  N . If we write a point z  N  in the form z = 
(z, z)  N x N -, then the following equations determine the intersection of the 
spheres { I I111 =  +/a{ I I1 - 11 
2 
, + IIull = = (1 + a)=, 
(z - 3)  
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<2a(n-1) l +()2-t 2  dt 
1. 
= 2 o(,, - 1) (1 +d)n (1 -u2)«u, 
where 0 --  It is clear that 0 > 0 for 3 small euough. The explicit condition 
-- (1+6)  
for  is  < -11 XE esIilnaIe Ihc flmction (1 - u2) bv its maximal value 
-- 35 
on the interval [0, 1], and we obtain 
2(n-1) (1+6)"(1-02)(1-0) 
 2 (n - 1) (1 + 6)(1 - 0). 
Since a short cal«ula.tion gives that 1-0 N 5d again for small d (d  2/35), we get the 
desired estimate. Finally, we finish the proof bv putting 0 = rein{ -11 2 
35 ' 35 } 
35" 
} introduce the following notation. Let B = B[z, ri be a ball. The symbol 
(1 + ) = [,(1 + 
denotes the enlarged ball with the saine center and (1 + a) times bigger radius. 
shall be usiug both symbols (1 + )B and B[X, (1 + )r]. 
The next lemma contains the key estimate needed in the proof of Proposition 1. 
Lemma 3. Tbere is a number o > 0 sucb that or ever r > O. eve amil 
  (r) of disjoint balls of radius v. and eve finite-dimensional subspace U C H. 
the following estimate holds: 
1 
u((1 + a)Bo  {(1 +a)B I B eu, B/Bo})  (1 + )dimUu 0 
provided that 0 <   o and Bo  u. 
Pro@ Let Bo  u be fixed. Without loss of generality, we assume that Bo h 
center at the origin, Bo = B[0, roi, say. Let d > 0 be such that 2d < (- 3). 
Then. bv Lemma 1, we sec that the ball (1 + d)Bo is disjoint with 
{B[x,(l+)rx] IB[x, rx]eu, 2rx ro or 2r0rx}. 
Accordingly, the only relevant balls in u that may interfere with the (1 + )Bo 
are those of radii comparable to r0. ç denote the centres of such balls by 
C 
Note that for the ball B[x, rx]  u with x  C we have }r0  rx 5 2r0. 
We have to estimate the measure of (1 + a)BoGec B[x, (1 + a)r]. Since 
xEC xEC 
we shall look closer at ea«h intersection (1 + J)B0  B[x, (1 + J)ræ]. 
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Let x E C. First note that 
o +  _< I111 _< (1 + )(o + ). 
Also, r < 2ro. We show that 
(6) B[x, (1 + 6)rx] C B 3ro iï,2(1 + 36)r0 . 
To see this, let y e B[x, (1 + 6)rz], i.e., []y - xii G (1 + 6)rz. Then 
x 
I 
3r0 
, 
_< (1 + 6)r. + IIIxll- 3fo. 
If 3fo > IIx[[, then the above calculation finishes with 
<_ (l+6)r+3ro-(ro+r)=2r o+6rz 
<_ 2(1 + 6)ro < 2(1 + 36)ro. 
If, on the other hand. 3ro G ]]x]], then we proceed  
 ( + ). + ( + )(o + .) - 3o 
 5(1 + 6)ro - 3ro < 2(1 + 36)ro. 
It follows immediately from (6) that 
Now let  = dilnU for short. If, moreover,  N &a from Lemma 2, then we 
obtain the estimate of the intersection on the right-hand side of (7): 
((1 + a)Bo  B[x, (1 + a)r]) 
X 
  ( - ) 0 ( + )- 
- u  (] +) B[0.o]. 
() 

Hence 
(8) 

< o(n- 1) 
10 -T (1 + 6)' 6  Lf,B[O, ro] .cardC. 
- 
What is missing now is some control over the cardinality of the set C. Fortunately, 
for our purpose we shall hot need any hard estimate. The sufficient upper bound 
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for card C follows frolll thc COlllparisoll of certain volumes. To this end, recall that 
for ail x G C 
Ilxll < (1 + ()(r + o) and 
_ ro  rz  2fo. 

(9) 

U B[x. r] C BI0. (5 + 35)ro]. 
1 . 
Also, B[x,r] D BI.r, 'o] for x E (7. eombining it with (9) we get 
B[x, ro] car(IC  B[O.(5 + 36)ro]. 

Thus 

( 
cardC < 10" 1 + g < 10'(1 + 6) '. 
Using this estimate in (8) we have 
< (n- 1) 10 (1 +6) 
- 
(,- 1) . ):-  .[O,o]. 
= m  (1+ 
(,) 
(n--1) 
Since (n) fornoe, thereis6 >0suchthat 
(n- 1) . 1 
m  (1+6) 6<- 
for all n G N and 0 < 6  6. With this choice of 6 one h 
1 
(1o) ((1 +)v  U v. (1 + )1)  (1 + )v. 
1 
To complete the proof, we put o 
then by (10), 
n ((1 + )B0 X U{( 1 + 
=(1 + )Bo- ((1 + )Bo  U B[x.(1 + )r]) 
1 1 
 (1 +6)Bo - (1 + 6)Bo = (1 +6)Bo 

and the proof is finished. 

Ifnow 0<6<_6o, 

V(e associate with every Bo  3u the set 
DBo : (1 +6)13o \ (Bo U U{(1 +5)B I B  d, 13 #/30}). 
Then, obviously, {De ] B E 3u} is the disjoint system of subsets in U. 
consequence of Lemnm 3 is the following estimate of the measure of De. 

Olle 
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Corollary 1. Let 50 > [) be as in Lemma 3 and let U C H be a finite-dimensional 
subspace. Then 
çUDBo >_ ((l + )dinlU-1) çuBo 
for every 0 <  <_ o and every Bo  u. 
Proof. Since 
o  o = ( +6)o {( +6) I  e ,  # }, 
we obtain by usiug Lelnlna 3, 
1 (1 + )dimU UB0" 
The sers DUo and B0 are disjoint. So u(D8o U Bo) = uD8 o + uBo, and the 
statement follows by rearrangement. 
Now we shall estimate the so-called packing density of the familv u in r. Since 
if is a finit.e-dimensional subspace of H, we ident.ify it with , n = dira U. Ve 
put 
the n-dimensional cube in {r of side 2k. With this notation we can state the 
followiug 
Lemma 4. There is o > 0 such that for every finite-dimensional s,ubspace U   
aad every r > O. 
{.<.nU. I} < 
o0< U'h(l+)--l>0l 
PWO. Be   (r) be arbitrry mie le  > 0 be  in Bemm 3.  denoe 
kr R tle [amilF o[ al] blls in g sur at te (1 + ) enirenmnt o[ B is still 
contained in the cube Qe, 

BE93u 
(11) <_  ,B + (Qk \ Qk-»(+) 
provided that k > 2r(1 + ). By Corollary 1, 
1 
for n with (1 + ) - 1 > 0. Also, D C Q for any B  N. Since the sers D 
are disjoint for different B's, we may sure up the estimates in (12) to get 
1 
  D < y Q. 
(13)  B  }(1 + 51'- 1 eu - (1 + - 1 
B 

Then 
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Lookiug back to (11) ouc bas 

2'.Q n U,u) < 

Since the expression on the right-hand side does not depend on , the saine estimate 
holds truc also for the supremum over ail   (r). Hence 

+limsup[1-(lt._. 2r(l+5)k )] 

aud the lemma is proved. 

The straightforward reformulation of the statement of L.emma 4 is the following: 
For any cubc Q c l,  N', 
(14) limsup sup{ (QUu) I   (r)} < 1 
r0 Q - (1 + 5) - 1 
for any 0 < 5  5o and ail n  N suciently big. 
Until now we have used only Lebesgue meabure. The next (ey) lemma allows 
us to get the estimates for auv other mesure absolutely continuous with respect 
to the Lebesgue mesure. 
Lemma 5. Let f  LI(N n) and let (K), r > O. be a system of measurable se 
Iç C  satisfying the following condition: 

There is a > 0 such that for every cube Q c IR ', lira sup 

Then 

f 
limsup / f d _<  Il/lit,- 

2'.(Q n/,) < o. 
2'. - 

Ig(x) - g(Y)l < e 

for an3' x, y  IR ' satisfying IIx- yll < 5- 
Let Q c IR ' be a cube containing the support of 9- We partition the cube Q into 
a finite fanfily/9 of subcubes of diameter at most 5, and then we choose iii each 

Pro@ Let e > 0. There is a continuous function g" IR n  IR with compact 
support such that IIf - gllL < e. Further, bv the uniform continuity of g, there is 
5 > 0 such that 



VITALI COVERING THEOREM IN HILBEIT SPACE 
point xp 6 P, for example the centre. Now 

(15) 

g d.n 
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By the assumption we can choose r > 0 small enough to guarantee that 
for ail P Ç . Then the last sure in (15} can be estimated by 
PEP PEP 
p6p dP 
s (« + ) (11/- 11, + II/11, + 
Combining this estimate with the estimate in (15), we obtain that 
rO J" 
/ 
Since e > 0 is arbitrarily small we couclude that 
lim sup f f dn  «ll/ll, 
rO JK 
which completes the proof. 
Proof of Propositio 1. Let e > 0 be given. We choose  E (0, 0] such that the 
conclusion of Lemma 4 holds truc. Also, let  E N be lge enough to satisÇ 
(16) !(1 + )-n_ 1 - a 
2 
By assumption, there is a finite-dilnensional space U, dira U OE . such that  is 
absolutely continuous with respect to the Lebesgue measure u. We denote 
d 
S- 
For every r > 0 there is a () E (r) such that 
(17) ,u(U ))   
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We put K,- = U ), r > 0. For this choice of Kr the assmnption of Lcmma 5 is 
satisfied: Let Q c U be a cube. Then by Lemma 4 in the fom (14) and bv (16) 
we have 
lin, sup U(QU)) <lilnsup su t ) u(QUu) < . 
o uQ - o e() uQ - 3 

So Lemma 5 imt)lies that 
}in, s,,p/,u (U ï' ) 

f 
= lim sup I 
-o Jouir ) 

f d..u < -. 
-3 

In combination with (17) we g('t 
2e 
li,,,s,,p s,,I,{/,u (U,') I   (r)}  . 
It follows that therc is vo > 0 suçh that fi)r ail ) < r  vo we have 
(') ,,,,{,,(U ,)    (lr)} . 
r0. By condition (iii) in Proposition 1 and (18), 

<_ L {,,(u.)   r} ,,,,  L«+. :. 
Since this is true for ail   (r) we mav conclude that 
,,{,(u)    ()} . 
provided that 0 < r  r0 and Proposition 1 is proved. 

It moEv be of solne interest to lnake the following final relnark. Although the 
classical version of the Vitali Covering Theorem fails for e.g. ail infinite-dimensional 
Gaussian measures, there is still a weaker statelnent of the covering type which holds 
true. The validity of the Differentiation theoreln is, in fact, equivalent to such a 
weak covering theorem. For details see e.g. Hayes and Pauc [2], or the deep paper 
of M. Talagrand [7], where this connection is treated in considerable generality. 
Based on the already-mentioned positive differentiation result [8] for some class 
çof Gaussian measures we have the following: 
Given "y  Ç, 1 < p < oc,  > O, ad Vitali system 12 on a set A 
in a separable Hilbert space, there s a countable subsystem S C ç 
such that 
(i) -y(A \ U S) = O, 
Condition (il) means that instead of disjointness we are only able to make the 
overlap of sers in S arbitrarily small in the given L p norm. 
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ON IDEMPOTENTS IN REDUCED ENVELOPING ALGEBRAS 

GEORGE B. SELIGMAN 

ABSTRACT. Explicit constructions are given for idernpotents that generate ail 
projective indecomposable modules for certain finite-dimensional quotients of 
the universal euveloping algebra of the Lie algebra sg(2) in odd prime charac- 
teristic. The program is put in a general context, although constructions are 
only carried through in the case of sg(2). 

§1. INTRODUCTION 
We consider the Lie algebra se(2) with basis e,f,h, relations [ef] = h, [eh] = 
2e, [fh] = -2f. A rednced enveloping algebra for the Lie algebra 9 = sf(2) over 
a field F, assumed algebraically closed and of odd prime characteristic p, is deter- 
nfined by three parameters , p, oe  F. It is the algebra  defined as the quotient 
of the universal enveloping algebra  = H(sf(2)) by the ideal generated by the (cen- 
tral) elements h p -  - , e p - p, fP - oe of g. The dimension of  is pa. We identify 
e, f, h with their images in  whenever there is no great danger of confusion. 
The subalgebra F[h] of  is semisimple; since the minimum polynomial XP-X- 
 of  factors as H (X-E-), where T  F is fixed with EP-T =  (we fix vE = 0 if 
 = 0), we let g(X) - x'-x- for each a  Fp. Then h(a) = g(T+a)-g(b) is 
a nonzero idempotent in F[h], and these h(a) form a complete system of orthogonal 
idempotentsin F[h]" h(a)h() =Oifa  ,  h(a)= 1, b(a)h= (T+a)h(a). A 
generalization in thc ce of the restricted enveloping algebra, yielding a complete 
set of idempotents in the cnveloping algebra of a Cartan subalgebra, w given 
by Nielsen IN] in 1963. In that thesis, Nielsen also gives generators for minimal 
one-sided ideals in the restricted enveloping algebra of 
We propose to refine this set of idempotents to a complote set of primitive 
idempotents in . Ve work in the comnmtative subalgebra of  generated by 
h and e f, and in the commutative subalgebra  of  generated by h, e f, e p and 
fP. (The subalgebra  will only be involved in 5, where we shall recall that 
certain elements of  satisfy relations, such as belonging to the ideal  in  
generat.ed by e p - p, fP - v,h p - h - £. For exmnple, the elements h*() = 
ga(w+)-ga(h), now regarded as elements of , still satisv  h*() = 1 6 , 
with h*(a)h*(Z)- 5, (ff) in the ideal in  generated bv h p h A, as is 
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The kcy to the construction is the following observation, easily proved by induc- 
tion: 
Lemma 1. For any set of i or more indeterminates X,...,Xk, let si[X,..., 
be the i-lb elern, entar'.q symmetric function (s0 = 1). Then for every positive inte9er 
j, and each c E Fp, 
j-1 
(1) ()el  =  sk(+ - + , e( + - + 3) ..... (j- )(+ - + j))(()«l)-k. 
k=0 
We indicate a proof, with h() replaced by h*(), and working in A. that both 
si(les of thc relation are in fl a.nd are congruent lnodulo the idem in fl as above. 
Thc relation in  then follows. For j = 1, there is nothing to prove. If one writes 
h*(c)eJ+l fJ +1 = h*(o)eJ fJ«f +  h*(a)eJ f[eY]f )- 
k=O 

and ses 

[efl = h,h*(c)e2 fh f - = h*(o)he f  + 2(j - k)h*(c)eg f , 
one deduces inductively that ail h*()ef  are in . Furthermore, ail ef  are in 
. so thal h*(«)hefJ is congruent, lnodulo our ideal  in , to (ç+ a)h*()ef . 
Thus 
t,*(a)d+f j+ m *(a)eJ fJef + j(a + r + j + 1)h*(a)e j ri(mode), 
giving the necessary indu(rive step. When j = p (still working in ), we have 
h*()ePf v  ph*(a)(lnod3). If b  h*(a), then h*(a)b  b(lnod3), so that 
p--I 
h*(a)   s(r + a + 2,2(r + a + 3) .... ,(p- 1)(r + a + p))(h*(a)ef) v- 
(mod3). 
In the holnOlnorphic image in  of . the image h() of h*(a) is the unit 
element for the idem it generates, and the element h(a)ef of this ideal satisfi the 
polynomial equation f(X) = 0 relative to the Ulfit element h(a), where 
f(x) =  (x + i( +  + i + )) - . 
Fp 
Changilg the variable here to } = X - )2, we bave 
2 + i)2) _ u. 
iFp 
Now it is a straightforward exercise to verify that. for indeterlninates 1" and Z 
over Fp, 
 (v +(i+ z))= 0" + (-)) +(z-z) . 

Thus 
(2) 

() = )(r + (_))2 + (-;ee -)2_ , 
A2 
= g(.- + (_))2 + (_ ). 
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Hcre g'(l') = (-1) 9-+Y + 1, so g(') is semisimple if and onlv if no element 
_fil2 (0 / G Fp) is a foot of 9('). One easily checks that 
g(-/3 2) = _;2 + / _ ; + ¼- _ , =  _ . 
Tha.t is, g(') has repea.ted roots if and only if ) = 41tv. We refer to ) - 4/v a.s 
the discrimilm.nt of °A. 

§2. Trie SEMISIMPLE CASE 
When the discriminant is nonzero, f(X) has p distinct roots 0 ..... 0p. Lagrange 
interpolation may be applicd once more to obtain h() (or h*(), modulo ) a.s a 
sure of p elements ill F[ (),  (o)ff] that are nonzero orthogonal idempotents (with 
the sa.llle holding with b(o) replaced by h*(), if we work in . modulo ). As 
 rms over Fp, Olle obtains p2 orthogonal idempotçnls in . with smn 1. Typical 
for thcse is an elenlent E(n. 0) = h(n)q(b(o)ef), whcre q(X) is a polynomial of 
degree p- 1. depending on o and 0, with E(n. Oi)h(a)ef = OiE(n. Oi) in  (or 
with a correst)onding congruence in . modulo ). 
Under these cirçmnstances, it is easy to sec that every irreducible %module  
has dimension at lcast (indeed, equal to) p: If one of p, u is nonzero and if v is an 
eigenvector for h in , then so are all ve a. vf t', and one of these fmnilies yields 
eigenvectors for p distinct eigenvahles. If t = 0 = u, t,h = h-v, we lnay smne 
ve = 0. Thon the usual treatment of t(2)-modules shows that if the first integer 
k > 0 with t'f k = 0 is lcss than p, thon   Fp and A = 0, a contradiction. Tlms all 
vf t, 0  k  p, are linearly independcnt. (More detail on irreducible modules will 
be nceded bclow; for a more thorough and extensive study of irreducible modnles 
in all reductive cases, sec [JCJ].) 
It follows that the right ideals E(a. Oi) have dimension at least p. But there 
are p such ideals, and the algel)ra . of dimension pa, is their direct sure. Thus 
each E(a. Oi) is a minimal right idem in . raid  is a semisimple algebra. From 
general principles, the modules E(a, Oi) group into p blocks, each consisting of p 
isomorphic -modules.  dcternfine conditions for isonorphism in terres of the 
parameters a and 0. 
First, the promised details on irreducible %modules: If  is such a module, then 
h and er commute in their açtions, so have a common eigenvector v. As above, if 
tf ¢ 0 or u ¢ 0, then ail ï'e j or ail vif, 0  j < p are linearly independent. One 
readily verifies that their span is stable under the action of e, f, h, so is equal to . 
Ift=u=0, thenve  ¢0, ve + =0 forsome r, 0 r <p. Letw=ve . Then 
, t'f ..... "u'f p-1 form a basis for l. Haying fixed r (necessm'ily an eigenvalue of 
h), the eigenvalue of er to which a r-eigenvector for h belongs determines l up to 
isomorphism. 
In the present case, E(a, O)ef = E(a, O)h(a)ef = OE(a, 0), and there is some 
e j or fa (possibly both) with E(a, O)e j or E(a, O)f  an h-eigenvector of eigenvalue 
v. The commutation relations then yield that the corresponding eigenvalue of er 
is 0- (v +  + 1). (Independently of whether we use E(a,O)ef or E(a,O)f: for 
instance, if 0 ¢ E(a, O)f  is an h-eigenvector belonging to the h-eigenvalue ç, then 
2k = a lin Fp], and, inductively on m, E(a, O)fef = (0 - mç - ma + m(m - 
1))E(,O)fm.) 
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Aceordingly, we bave E(o, O)°A - E(, o')gJ if and only if 
a = 0'  fi 
(3) 0-ça(r+ç+l) -ç(z+g+l). 
It may hdp to clarify this stalement bv noting that if 0 is a root of f(X) = 
 (X + i(z + a + i + 1)) - p.u as before, and if we make the change of variable 
iFp 

c*(r+ +1)- + +1), 
X = X'+3  (T  

t hell 
fl)('r +fi + (i + - - -) + 1)) - #v 
f(X) = f*(X') = H (X' + (i + ct _ 
2 
2 
-6F o 
= (X'+j(r+fl+j+l))-pv, 
jF 
so lhat 0' =0- (r +  + 1) + (r +  + 1)is a foot of f*(X'). For each  #  
in Fp, there is OlW such 0'. Thus, for given a, ail E(a,0) are nomimmorphic 
%modules, while for each pair a,  (a # ff), the isomorphism cluses of modules 
E(a, 0) and E(fl, 0') are lhe saine when the relation (3) holds. 

§3. THE DEGENERATE, NON-RESTRICTED CASE 
When ,X 2 = 4pu, 
f(X) = (X - 
1 
1 
2 )2)((x- (  )2) + (_1))2 
(by (2)) is the milfinmm polynomial of h(o)ef relative to the unil element h(o). 
The roots in F of Y + (-1) a = 0 in F are lhe negatives of nonzero squares 

in Fp. So 

f(X) = (X - (7- + e, + 1 ( r + a + 
2 )) 1-[ (x - 2 1)2 +.)2. 

The following lemma, whose proof is lefl as an exercise, gives a resolution of each 
h(c) into orthogonal idempotents: 

1-[ (Y+ 7) 2, and, for  C F*p 2, let ge(Y) = 2Y(Y- 

r + a + 1)2) = g(}-)). 

g((I')g,(Y) = 5(vgv(Y)(mod f(Y + ( 
2 
t onows tat the elemem.s E(, O) = 0(()), t(,fi) = (()) where 
(r+a+)2), form a set of P+ (nonzero) orthogonal idempotents 
L(x) = g.(x-  
in F[h(a).h(a)ef] C , whose sure is the unit element h(a). (When h(a) is 
replaced by h*(a), the corresponding relations hold (mod,) in the subalgebra 
f[h*(a),h*(a)ef] of the conmmtative subalgebra J of tt().) As ct nms over 
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we obtain a family of p(p+t)2 orthogonal idempotents E(c,/3) in F[h, e f] C 9.1, whose 
SIllll is 1. 
To sec that these idcmpolents are primitive, one may reason as follows: Each 
E(c, 0), c E Fp, generates a nonzero right 9.l-module. If we assume not all of ,, p.. , 
are 0, then as before, this module bas dilnension at least p, and dimension exactl) 
p if and onlv if it is irreducible. H'e defer the "restmcted " case , = p. = oe = 0 to 
the nezt section. When/3 E FÆ 2, lefl-nmltiplication bv h(c)ef - (-++1)2h(c) + 
th(c) is a nonzero nilpotent 9-l-endomorphism of E(c,/3)A whose image lies in its 
kernel, and where each of thc kernel and cokerncl bas dinension at least p. Thus 
E(o,/3)9.1 has dinlension at lee, st 2p, and the dimension is exactly 2p only if both 
the image and kernel are irreducible. In that case the endomorphism above induces 
ml isomorphisnl of the cokernel onto the kernel. The SUln of the dilnensions of all 
P__7A. 2p = p3 = dira 9.1. with equalitv if and onlv 
E(c,/3)A is thus at least p2 + p. 2 .  
if each E(c. 0) is irreducible and each E(c, /3)9.1. /3  F 2 has dimension 2p. Prom 
the above, the latter modules are indecolnposable. We bave the 

Theorenl 1. If ,2 = lpu, hot all of ,k, p. u being equal to zero. riz a solutzon T 
of X p- X - , = 0, w-ith r = 0 if , = 0. Form elements E(a, rl) for all l  F 
p, 
a  Fp, as above. Then these P(P+) elements of F[h, ef] C  form a complete set 
of primitive id«mpotcnts in °d. The right ideals E(c. O)°A are isomorphic irreducible 
°d-modules. The right ideals E(a,/3),/3  F*p 2, are (projecti-ve) indecomposable 
9.1-modules of length 2. each with isomorphic composition factors. Two modules 
E(a, ri)9.1 and E(o/, l')°A are isomorphic if and onlg if ! = 1'. 

Pro@ Only the assertions about isolnorphisms remain to be proved. Here we may 
assume p. 7 0: the argument for oe 7 0 is analogous, and both cannot be zero 
because ,2 = 4#,. (Some isomorphisms, e.g., those of ail E(c, 0)9.1. follow from 
general theory, but our identification gives a little more detail.) 
An element of the irreducible module E(o. 0)9.1 of h-eigenvalue r is lï(o. 0)e s, 
where 2j represents -c(modp), and 

E(o,O)e.ef = (( 

r+c+l 

2 

)2 +j(r + c + j + 1))E(c.0)e . 

But j(r +(+j + 1) = - (r +(- + 1), so that (++'-2)2 +j(T+(+j + 1) = (---)2, 
independent of (. Thus all E((, 0)PA are isomorphic. 
Similarly, the quotient of E(c,/3)°A, for/3 E F 2, by its unique maximal submod- 
ule has as h-eigenvector of eigenvalue r the coset of E(c,/3)eJ, 2j = -c as above, 
and 
T+l 2 
E(c,/3)def =_ (-/3 + () )E(c,/3)d, 
as before, here modulo the maxinml submodule. By [C-RI, Theorem 54.11, if N 
is the radical of A, E(a,/3)9l is the unique nmximal sublnodule of E(c,/3)9.1, and 
E(c,/3)9..l and E(cg,/3')9.1 are isomorphic if and only if 

(,/3)/E(c /3)91 - t(a',/3')/E(.',/3')91. 

We have just seen that this last isolnorphisnl holds if and only if/3 = ff'. This 
completes the proof. [] 
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,4. TrIE ESTRIC'I'ED CASE 

Now let ,X = p = u = (I. In this case, the proj¢'ctive indecomt)osable -modules 
are m,,re compliçated, lmt their stru«Im'e is well known (see [P]). There is the 
irredueii»le Steinberg module, gcncrated by an element 
and basis v. vf ..... vfP-: and there are p - 1 modules, each of dimension 2p and 
length 4. We assign to the Steinberg module. "{ilp_", the parameter p - 1. and 
paranmters 0 ..... p- 2 to the remailfing modules, as follows: 
The module 9)lt. has generator ,, with ut.h = ku.. To give flrther relations, 
and fin" future rofl'n'n«e, we de,fine, for a Ç Fp, res(n) to be the ordinary integer 
j.O  j < p, whose residu¢, class (m¢,dp) is n. Now let k' = p- 2 - k (¢ kmodp) 
mM h,t rt. = res(). Let z,, 
Th{,n vt.,  0 bnt v,e = 0. and 
pk, f"« -1 = (-l)"«-l(,.t. _ l)2ake. 
A basis [)1 331 k consists of t}le e]Olllçllts {utff,v,ff}.O  i < p. The unique 
maximal submodule of 93lt. is (,,} + (.rf+), of codimension k + 1. 
lit,re T = 0. and again, we havc 
given as in 3. The algcbra '3 is a Frobenius algebla [Ber], indeed a symmetric 
alKobl'a [SI. So ihe nmltiplicity of 9)lt. as an iudecomposable summand of  is equal 
to lhe dimension of the quotient of llt. bv its maximal sulmodule, or  + 1 ([C-], 
Theorem 61.13). Il follows that the ninn])er of indecomposable summands of  is 
9 
and therefore that ail our (nonzero) P(+) orthogonal idempotents are prinfitive, 
and ail E(a. fl)gl are indecomposable. 

Theoreln 2. II'hen A = I = e = O, ail E(a,0)*2t are zsomorphc for a  Fv, 
and are the (irredu, cible) Steinberg modu, le .9v_  . For 1 < j <  . tbe modu, les 
E(a, j2)9./are indeconposablç, witb E(c, j2)°A isomorphic to e(-2j-) or ffR2j-, 
ccodi,9 . j < ,-s(-_ 
complete set of primitive id«mpot«nts 

Pro@ The last assertion has |)een established. If r is luininlal vith E(a. O)e +1 = O. 
then 
I = E(c.O)er+l f = (a + 2r + 1)E(a,0)e . 
9 
So o  -2r - l(modp), and w = E(o. 0)e  bas u' = -u,, u'e = 0. It follows that 
u', u'f .... , ml p-1 form a basis %r an irreducible submodule isomorphic to 
with 
,,'/ = «(. 0)«/ = (,')«(,0) ¢ O. 
Thus E(o, 0) = (w)  lp_ 1. 
For 1  j  , we know that E(a,j)  l for some k, 0  k  p- 2. and 
that in anv such isolnorphism the element E(o. j) callllOt correspond to a member 
of the proper submodule (vk,) + (uf +} of k. Thus E(a,j ) must correspond 
to an element i,k,f + çut.f t where 1 # 0. t  k and both k' - 2s and k - 2t 
represent o(modp). From the last remark, we have that k' - 2s equal to one of 
k-2t, k-2tp. Ifk -2.s=k-2t.thenk -kiseven,contrarytok+k =P-2. 
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If h'-2s = k-2t+p, then p-h-2-2s = h-2t+p, 2t = 2k+2s+2, and t > k, a 
contradiction. Thus p - 2 - 
and t < s. 
Applying fv-l-t to our expression corresponding to E(a, j2) gives 
E(o,j2)f p-l-t = elzkf p-1  O. 
That is, ukf p- belongs to the eigenvalue a - 2(p - 1 - t) of h, and is annihilated 
by f. Its eigenvalue is evidently equal to k - 2(p - 1) _= -k'; so a + 2 + 2t = -U 
in Fp. 
In other words, for the nonnegative integer m such that E(a,j2)f ' : O. 
E o A2¢m+l 
( , j jj = 0 determines U by U = res(2m-a), and thereby the isomorphism 
class of E(a, j2)2t: E(a, j2)2t  9Jtk, where k = res(a - 2m - 2). To deternfine this 
"rn", we invoke the following: 

(x) = 2(x - j2)x 

Lemma 3. Let 

H(X + i2)2. l<j< p-1 
-- -- 2 
i=1 

k-1 

Let r(c,j) be the largest value of !; such that fc,,k(X) = [I (X + i(c + i + 1)) 
i=0 
(«+)2) Then E(c,j2)°A is the projective indecomposable module 
divides gj: (X -  . 
res( + 2n( ,j) ) " 
Proof of Lemma 3. With ri(a, j) as defined, 
.+1 +1 2 
(x - )) = ,(x - () 
where hj(X- + 2) + 2 
() is a product of factors involving onlv X + k 2 
p-l. a-I 2. j2 a+l)2. 
() , - - ( In , we have therefore 
l<k<,X- andX 
a+l 
E(,j) = h(()«I- ()h(a))L,.(.,)((a)«I) 
= j((a)ef- (@)h())h(a)«"("')l "("'), 
by Lemma 1. Now h(a)ef commutes with ail h(a)ef , and the proof of Lemma 1 
involves showing that 
h(a)e f%f = h(a)e+ f + - k(a + h + 1)h(a)e f . 

Thus 

and 

E(a,j 2) = h(a)e'(«,J)f'(«4)(hj(h(a)ef - (---)2h(o))) 

h(c)e '(«0)f'(«'j)h(a)ef 
= h(a)e'(«'i)+lf ''(d)+l - n(,j)( + n( + n(, j) + 1)h()e(«d)f («d). 
(x-()). 
By definition, X+n(,j)(+(a, j)+l) is hot among the factors ofgj  «+ 
Thus the coefficient of h()e(')f (mj) will hot be zero when E(,j 2) is ex- 
panded in terres of the form h()ef i, while the other values of "i" that oc- 
cur will ail be greater than n(,j). Accordingly, E(mj)f p-(«'j}- ¢ 0 while 
E(, j2)fp-(«,) = O. The lemma is proved.  
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To COml)lete the proof of thc theorcni: Note that n((,, j) is the first integer -n with 
n( «+n+ 1) =j2-/--+A2", 2  , so ,((,j)=-(-21)+j(m()(lp). Ifj < rcs(-(--)«-Pl ) < 
p- j, thon n(e,j) -- rcs(-j c+1); otherwise, n(c,j) = res(j c+l). The 
2 2 
rcmnining assertion of thc thcorcm follows. 

§5. SOME (]ENERALIZATIONS 
The rcmarks of this section al)I)ly gcncrally. Thc cae where the underlying Lie 
algebra is s(2) will bc ot)served, from earlier constructions, to satisfy the conditions 
imposed on thc klemt)otcnts. Al)l)lication to that case enables us to recover, in 
somcwhat more explicit form, some results of Christopher Bcndel [Ben]. 
H«rc $ is au ari)itrary Lie algebra of primc characteristic and finite dimension 
over a fiel(1 F, assumed algcbraically close(l (although further asumptions avoid this 
('onstraint). Let xl .... ,x i)c a i)asis for 0. br each xi, lct z bc a polynomial 
in xi that is central in (0), say of degrcc p"'. Fix scalars A ..... A,  F and 
a n(mncgativc intcgcr s. Lct  be thc i(lcal in ri(0) gcnerated bv the (central) 
pS 
clements Ti , 1  i  n, whcre T = z - As. Let ' bc the i(leal generated bv the 
T. Then  = (9)/ and '3 = ($)/oE are finite-dimensional algebras over F, of 
respective dimensions p»+E , and pE , ([J], Çhapter 6). 
SupI)ose we have fomd elcments e ..... et  [(9) such that: 
t 
i)  e = 1 iii 
ii) there is a COlnnmtative subalgebra  of (9), containing ail ej  and all 
T,'s, such that for ail j. k, eie - 5ie is in the ideal in  generated by 
the T. 
These conditions guarantee that the homolnorphic images êi of the e in  form 
a svsteln of orthogonal idempotents whose suln is 1. The algebra "r" previously 
used guarantees that the polynolnials in the h*()ef 3"ielding idempotents in the 
"" of earlier sections stisfy i) and ii). 
From i) and ii),  ei = 1. If eie -5ie =  wT, u'  . then 
j=l n=l 
«  -  =  '7 ç . 
p 
So the homomorphic images in  of the e are a system of orthogonal idempotents 
p pS 
whose sure is 1.  denote the image of ei in  by E. If ei  '. then « - e = 
pS 
E (+ - «)  '. « s  ¢    # 0   # 0. 
The central subalgebra FIT1 .... , T] of 
as generators. AI1 monomials Tï ... T  with some i OE p are in . XX order the 
remaining monomials {M} as follows: M < N if degree M < degree N, and the 
order is linear but otherwise arbitrary among mononfials of the saine degree. 
label by subscripts indicating place in the order. Thus 
1 = Mo < [1 
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Then the right ideal E in  has descending filtration 
ï : (ï) D... D )» D 0, 
where (E) : E EjAIq. 
qr--1 
In the right action of  (or of g()) on E]. ç lnaps each (E) into (E])+. 
So successive quotients in the filtration are right -modules. Let ç be the canonical 
homolnorphism of  into .  that of g() OlltO . SO that ç o  is the canonical 
homomorphism of g() onto 9. Then 
(ç  ')(«()) = ç() : (ç  ¢)(«) : (ç  ¢)(e) : ê. 
The kernel of ç I(E) contains (E))» So ç induces a homomorphism of 
(E)/(E) onto ê (as U()-modules, or 
r0 
Ker ç (by PoincafBirkhoff-Witt - see [J], Chapter 5). and 
r0 
 E3I, by lefl«nultiplying with the idcmpotents E. Thus the kernel of 
r0 
the induccd map of (E) onto ê is exacth- (E)» In particular, dira. 
For each r. 1 
= (0), by definition.) Thus 
dm((E])/(E])+t) S dira ê 
for every r. every j. Sulmning over r and j gives 
diln :  dim((Ej)/(E)+)  p dim = dim. 
yielding equality at ail stages. That is, we have proved: 
Proposition 1. With notation as above, the filtration (E) D (E) » ... D 
(Ej)p» D 0 of Ej by right ideals bas successive quotients which, as -modules, 
are all isomorphic to êj. 
Clearly the radical 9I of the algebra  contains ail ¢(T,). So E]/Ejgl is 
a homomorphic image of Ej/(E)2, and the submodules of Ej/EjgI are in 
correspondence with the set of submodules of ê] containing êç(N). Because 
ç(N) is a nilpotent ideal. êjç(N) # g. If êj is a primitive idempotent, êj has 
a unique maximal submodule ([C-RI, 54), a fortiori a unique maximal submodule 
COltainilg 
is the radical of . Thus E/E]N has a unique lnaximal submodule. Because 
every lnaximal submodule of Ej must contain EjN. E h a unique maximal 
submodule, and Ej is in turn a prilnitive idempotent. The converse is clear: If 
is primitive, Ej has a unique lnaximal submodule, which must contain (E); 
so êi has a unique maxilnal submodule. That is, 
Proposition 2. The idempotent ê] is primitive in  if and only if E] is primitive 
in . 
To apply the above to sf(2), with z{ = h p - h, e p, fP and the elements {ej} that 
are polynomials in the h*(a)ef  in 1-4, so that  is the quotient of g() by the 
ideal generated by the (h p -- h -- A) p . (e p -- p)P, (fP -- oe)P, we filld that the images 
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P" al'(' tt SVS|('III of primitive orthogona| idempotents of sure 1. In the 
in  of the c 3 . 
semisimt)le case, each of the corresI»onding projective indecomposable 3-modules 
has a. COml»OSii(m series of length pas with quotients all isomorphic o the irreducible 
9A-module êj°A. In the non-restricted case with A 2 = 4poe, here are p projective inde- 
composable lnodules with composition series of length pas and quotients isomorphic 
to the E(a, 0)°A(c  F), while the remaining p.i.m.s have composition-length 2p a*, 
composition-factors ail isomorphic to those of E((,/5),/3  F 2. These p.i.m.s 
adroit a filtration where successive quotients (pa, of them) are ail isomorphic to 
In the restrk'ted case, there are p prinfitive idempotents with generated modules 
having composition series of length pa, and the Steinberg module as quotient. The 
remaining  primitive i(|empot.ents each yie|d p.i.m.s of length 4p as, dimension 
2p 3s+, with filtrations having quolients ail isomorphic to the E((ï,/5)9A as in §4. 
These conclusions recover B('udel's results in [Ben] (§8). 
Il nmy be noted that, when F is algebraically close& every finite-dimensional 
indecomposable sg2)-module is a module for one of out alget)ras 3. with suitable 
values of A, tt,, u and s. 

[Ben] 

[NI 
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STABILITY OF INFINITE-DIMENSIONAL 
SAMPLED-DATA SYSTEMS 

HARTMUT LOGEMANN, RICHARI) REBARBER. AND STUART TOWNLEh 

ABSTRACT. Suppose that a static-statc fee(lback stabilizes a continuous-time 
linear infinite-dimensional control system. \Ve consider the following question: 
if we construct a sampled-data controller by applying an idealized sample-and- 
hold process to a continuous-time stabilizing feedback, will this sampled-data 
controller stabilize the system for ail sufficiently small sampling times? Here 
the state space X and the control space /_7 are Hilbert spaces, the system 
is of the form i(t) = Ax(t) + Bu(t), where A is the generator of a strongly 
continuous semigroup on X, and thc continuous time feedback is u(t) = Fx(t). 
The answer to the ahove question is known to be "yes'" if X and U are finite- 
dimensional spaces. In the infinite-dimensional case, if F is hot compact, then 
it is easy to find counterexamples. Therefore, we restrict attention to compact 
feedback. We show that the answer to the above question is "yes", if B is a 
bounded operator from U into X. Moreover, if B is unbounded, we show that 
the answer "yes" r«mains correct, provided that the semigroup generated by 
A is analytic. \Ve use the theory developed for static-state feedback to obtain 
analogous results for dynamic-output feedback control. 

1. INTRODUCTION 

Consider the following systeln with state space X and input space ( (both 
Hilbert spaces): 
(1.1) J:(t) = A.r(t) + Bu(t), x(0) = x °  X, 
where A is the generator of a strongly contimous semigroup T(t) on X and B is 
a bounded linear operator from U into X_, where X . is a certain extrapolation 
space with X '--, X-l. If the input (or control) operator B maps boundedly into 
the state space X, then B is called bounded; otherwise B is called "unbolnded" 
(with respect to the state space X). We give precise details of the set-up in Section 
2. 
Vve consider the following fimdamental problem: Suppose that the feedback 
control u(t) = Fx(t), where F is a bounded linear operator from X into U, is an 
exponentially stabilizing state feedback control for (1.1) in the sense that A + BF 
generates an exponentially stable strongly continuous selnigroup on X. A natural 
implementation of this continuous-time control u(t) = Fx(t) is to use sample and 
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hold, so that u is given instead bv 
(1.2) u(t) = Fa'(kr), t G [kr,(k + 1)r). 
tlerc r > 0 is the smupling period and t = nf, n = 0.1 .... , are the sampling times. 
The control (1.2) is called a sam pled-data fi'edback contlol and t he overall system 
given bv (1.1) and (1.2) is called a sampled-data feedback systeln. Intuitively, we 
would expect that for all suciently slnall Salnpling periods r > 0, (1.2) produces 
a stabilizing control fl»r (1.1) in the scnse that there exist N  1 and v > 0 so that 
(1.3) II(t)ll  '«-*ll."ll, v«"  x, vt 2 0. 
lntegraling (1.1) and (1.2) over one sampling inlerval [k,(k + 1)) and setting 
.r := z(k7) yields the following dis«rele-time svstem: 
.r+l = z where  := T(r) + T(s)BFds. 
h is straightforward to show that  is a bounded operator rioto X to X and hat 
(1.a) holds if, and «,llly if, a is power stable, i.e., I111  0  -   (sec Section 
2). Then he flmdalnental probl«m described above becomes that of deternfining 
wheher or hot  is power stable for ail sucienly small r > 0. 
Bv way of motivation, le us briefly discuss the finite-dimensional ce. Then 
,¥ = N n. I1 = N TM, A. B and F are real matrices of compatible formats and 
T(t) = e A. It is instructive to sketch a silnple argument showing iii this ce that 
the sampled-data feedback (1.2) is ind«ed sabilizing for (1.1), provided ha r is 
suciently snmll. Since u(f) = F() is a.n exponentially stabilizing static btate 
feedback conrol for (1.1), tire eigenvalues of t + BF have negative real pts, or 
equivalently, there exists a positive definite SVlmnetric matrix P satisfying 
(1.4) (A + BF)rP + P(A + BF) = -I. 
Next we observe using he power series expansion of the matrix exponential e m, 
that there exists M  0 so that 
II-I-r(A+BF)I I%M,r , Vre(0.1]. 
Then. invoking (1.4), it is easy to sec that there exist M. > 0 and T* > 0 such that 
zrAPAz - zrPz  2rzr(A + BF)Pz + Ah211oell =  llloell 2 , v e 
Therefore, for everv r e (0. r*), ATPA - P < 0. or, equivalently, A is power 
stable, showing that the salnpled-data feedback (1.2) is stabilizing for ail sufficiently 
small r > 0. A different pl'oof of this result can be found in Chen and ancis 
[1], where, using the power series expansion of the lnatrix exponential and the 
continuous dependence of the eigenvalues of & on r, it is shown that the spectral 
radius of A is less than one for all sufficiently slnall r > 0 (which implies power- 
stabilitv of &, for all suffciently small r > 0). 
Whilst this simple observation is easy to prove and well knowll in the finite- 
dimensional control literature, it serves to indicate that extending the above re- 
sult to infinite-dimensional situations is far from trivial. Indeed, neither the proof 
sketched above nor the proof given in [1] have lnealfingflfl generalisations to the 
infinite-dimensional case, because they both rely on the convergence of the Tay- 
lor series of e At in the operator-norln topolo', which extends to the infinite 
dilnensional case onlv if the generator of the semigroup is bounded, or. equivalently, 
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if the semigroup is uniforlnly contilmous. Systems described by Ulfiforllfly coutinu- 
ous semigroups are hOt general enough to encompass lnost interesting examples. 
It night appear that the case considered above of salnpling a static state or 
output feedback control, whilst mathelnatically significant and silnple to describe, 
is too restrictive for lnealfingflfl applications. However, results for the more practi- 
cally relevant case of sampling a dynalnic output feedback control tan be obtained 
from the state feedback case quite easily, as we show in Section 5 (in an infinite- 
dimensional context). Such salnpling of dynalnic contilmous-time controllers forms 
the basis for a so-called indirect salnpled-data design lnethodology (sec Chen and 
Francis [2] and Franklin. Powell and ç\.:orklnan [6] fol'a fuller description of this 
lnethodology in the finite-dilnensional case). 
1Recently there has been increasing iuterest in sampled-data control of infinite- 
dimensional systelnS; sec t/ebal'ber aud Townley [15], [16], l:{osen and \Valg [18] and 
Tarn et al. [20]. Froln these papers a t.rend emerges: if t.he underlying semigroup 
svsteln is analytic, or the input operator is |)ounded, then typically the finite- 
dimelsional salnpl,d-data result generalises to the infinite-dimellsional case; if the 
input operator is Ulfl)oundcd and the underlying Selnigroup is hot analytic, then the 
generalisation fails. It is hot surprising that almlyticity plays such a kev foie. In 
some sen,e, analyticity constrains the speetruln of the generator of the underlying 
semigroup so as to avoid auy losses of high-frequency inforlnation when sampling. 
We give a brief descriptiou of the results contained in this paper. In Section 2 
we introduce the class of infinite-dimellsional systems to which out results apply 
and we present some preliminary basic technical results to be used in the test of 
the paper. Iii Section 3 we consider the case in which the input operator B is 
bounded, but the selnigroup generated by A is not assulned to be analytic. We 
show that if the feedback operator F is COlnpact and the selnigroup generated by 
A + BF is exponentially stable, then, for sufficiently slnall 7- > 0, the sampled-data 
feedback (1.2) is stabilizing. A key element of the proof is the introduction of a 
new, equivalent norm on the state space with respect to which the operator A 
is a strict contraction, provided that r is sufficiently small. We also give a simple 
example which shows that the result is not truc if the Colnpactness assumption on 
F is removed. In Section 4 we consider the case in which the input operator B 
is unbounded and the semigroup generated by A is analytic. We show that for 
any compact feedback operator F, the operator A + BF (with suitable domain) 
generates an analytic selnigroup on the state space X, and if this semigroup is 
exponentially stable, then the salnpled-data feedback (1.2) is stabilizing for ail 
sutïiciently slnall r > 0. The proof of this result is quite involved and differs 
considerably from the argument leading to the result for bounded B. Instead of 
trying to estimate the spectral radius of A directly, we invoke fi'equency-dolnain 
stability criteria for the power stability of A, making repeated use of the analyticity 
of the underlying semigroup. Finally, in Section 5 we show how to use the static 
state feedback results of Sections 3 and 4 in the practically lnore relevant situation 
of sampling a dynamic output feedback controller. 
Notation: For a G IN, r > 0 and z G C we define 
The exterior of the unit disc is denoted by IE, i.e., 
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Let X and )" l)(, Ba)ael) Sl)a('es. The sl)açe of all bom)ded |tricar operators from X 
t.o )" is (h'n()t('d i)y (X. )); we set (.Y) := (X, X). bel A t)e an operator with 
domain an(l range in X. Th(,n D(A) (h'notes the (h)main of A and ç(A) denotes 
thc resolvent set of A. i.e., )he s(,i of ail s  Ç snch that si - A  D(A)  X is 
bijective mM (,I - A) -  (X). Thc sI)ectrmn of A. which is (he complement of 
ç(A), is deno(ed by (A). Let op(A) C «(A) denote the point spectrum (i.e., the 
set of M! eigcnvMues) of A. If S C X and 4  D(A) C X  )" is an operator, then 
.4[8 &'notes )hc restriction of A to S, i.e.. thc Ol)erator defined by As x  Ax for 
ail x  D(A]s) := D(A) S. For an open set l C Ç, H(IL (X.)')) denotes 
the set of Ml bomMcd analvtic (X, )')-vMu('d flmc)ions f dcfincd on an open set 
l I C  (d('t)cnding on f) snçh that l  l I is a discre(e set.  Clearly. each fimction 
J' tf(IL(X.))) a(hni)s an ex(ensi()n f  H(IL (X,)')) defined on l. For 
an ml)itrarv set A C C, we defin(" 
IIDA, çl open 

2. TtlE SAMPLED-DATA S ,STEM 
We start witl) a sin)plc l('mma whM) will be uschfl in the subsequent develop- 
lnelltS. 
Lemma 2.1. Let X, )" and Z be Hilbert spaces and let L " [0, 1] -- B(X,') be 
given. If limt-.0 L(t)x = 0 for all x  X and if K  B(Z, X) is compact, then 
(2.1) lira ]]L(t)Iç]l(z,y) : O. 
t--O 
Moreover, if limt-o L*(t)y = 0 for all y  '. and if H  3(, Z) is compact, then 
(2.2) lira 11/4L(011(x.z) = 0. 
t0 
Pro@ Assume that limt-,0 L()x = 0 for ail x  X. If K  (Z, X) bas finite tank, 
then it is easy to prove that (2.1) holds. In a Hilbert space a compact opertor 
can be uniformly approximated by finite rank operators, and a standard argument 
shows that (2.1) is true for compact K (Z,X). 
Assuming that limt-,0 L*(t).r = 0 for all x  X md that H  (]', Z) is compact, 
(2.2) can be obtained by all application of (2.1) to L*(t) and H*. [] 
Throughout the paper, X and U denote Hilbert spaces. Let. us consider the 
following sampled-data system with state spa.ce X 3nd input space U: 
(2.3a) :'(t) = Ax(t) + Bu(t), t > 0; x(0) = x °  X. 
(2.3b) t(t) = Fx(kr), 1,'r <_ t < (k+ 1)r, k (5 No, 
where x(t)  N. u(t)  U, r > 0 is the sampling rime, .4 is the generator of a 
strongly continuous semigroup T(t) oit X, F  N(X./if) and B  N(U, X_), where 
X_a denotes the closure of X tu the norlll II«ll- -- Il(Af- A)-axll (here Il" Il 
denotes the norm of X and A is anv element in o(A)). Clearly, X '--+ X_. and 
hence N(U,X) C N(U,X_). We say that B is bounded if B  N(U, X); otherwise 
we say that B is unbounded. The semigroup T(t) extends to a strongly continuous 
semigroup on X-l. The generator of T(t) on X_ is an extension of A to X 

1This means that either I =  or, if [ï ¢ fL then for each z G  \ .f there exists  > 0 
such that l(z,e)  (ç \ fil) = {:}. 
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(which is bounded as an operator from X to X-t). We shall use the saine symbol 
T(t) (respectively, A) for the original semigroup (respectively, its generator) and 
the associated extensions. With this convention, we may write A  N(X,X_). 
Considered as a generator on X_, the donmin of A is X. The spectrum of A 
considcred as an operator on X coincides with t.he spectrum of A considered  
an operator on X_; moreover, the point spectra of A and its extension coincide, 
including algebraic multiplicities of isolated eigenvalues, We refer the reer to p. 
123 in the book [5] by Engel and Nagel for more details on the extrapolation space 
The derivative on the leff-hand side of (2.3a) has to be interpreted in the space 
X_. To solve the initial-value problem (2.3), we define a flmction x recursively by 
(2.4a) x(O) = x ° , 
(2.4b) x(kr + t) = T(t)x(kr) + T(s)BFx(kr)ds, Vt e (0, ri, Vk e No. 
It follows from standard results in the theory of strongly continuous semigroups 
(applied to T(t) considered as a semigroup on X_) that 
x  C(N+,X) and x[[k(k+)l  C([ kr" (k + 1)r],X_), Vk  N0. 
Moreover, x satisfies the following differential equations in X_" 
&(t) = Ax(t) + BFx(kr), Vt e (kç,(k + 1)ç), Vk e N0. 
It is also clear that x given by (2.4) is the only function with these propeies. In 
this sense, the function x defined by (2.4) is the unique solution of (2.3). 
We say that (2.3) is exponentially stable if there exist N  1 and v > 0 such 
that ]]x(t)I]  Ne-t[[x°][ for all initial values x °  X and all t  0. To study the 
stability properties of the sampled-data system (2.3) we consider a related discrete 
time system. Let x be the solution of (2.3) given by (2.4) and set 

Then, by (2.4b), 
(2.5) xk+l = T(r)xk + [T 
JO 

x := x(kr). 

T(s)BFxkds = (T(r)+ STF)xk, Vk c No, 

where ST is an operator defined on U and given by 
(2.6) STu := T(s)Buds, Vu  U. 
Lemma 2.2. For any r >_ O, ST  3(U, X), and for any 0 > O, 
(2.7) sup IISrll(u,x) < " 
0<r<0 
Moreover, if F  N(X, U) is compact, then 
(2.8) lira IlSrrllm(x) = 0. 
T---0 
Pro@ By standard results from semigroup theory applied to T(t) considered as a 
semigroup on X-l, we know that STU C X and 
AS«u = (T(E)- I)Bu, Vu e U, Vv e (0, oc). 
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th'nce, fol" A E &(A), we ol)tain 
STu= (I - T('r))(AI - A)-IBu 
(2.9) 
+A T(s)(M-A)-IBuds, VuEU, 
showing that 'r  (U,X) and that (2.7) holds. Moreover, let £  (X. U) be 
compact.  sec from (2.9) that liln0 IISll =,  ll  e u, ,,a th,,s (2.8) 
follows froln an application of Lelmna 2.1. 
Introdu«ing thc Opel'ator 
(2.m) 
(2.5) tan be written in the form 
(2.11) '.+ = x, Vk N. 
Note tha.t bv Lelmna 2.2,   (X). ecall that an opcrator L  (X) is said 
The [ollowing simple result tan be obtained bv a slight modification of the proof 
o[ roposition 2.1 in «bal'bcr and Townley [16]. 
Lemma 2.3. For an r > 0. (2.3) is ezponentiall9 stable if and onl9 if  is power 
stable. 

Hence, fOl" the remail(ier of this pa.per, we stu(ty the power stability of A- in 
order to study the exponential stability of (2.3). 

3. BOUNDED CONTROL 
In this section we assume that the control operator B is bounded. We show that 
if the feedback operator F is compact, then expouential stability of the continuous- 
ti,ne semigroup generated by A + BF implies exponential stability of the sampled- 
data systeln (2.3) provided the salnpling time - is Slnall enough. More precisely, 
we prove the following result. 
Theorem 3.1. Assume that A 9enerates a strongly coztinuous sernigroup T(t) on 
X. B  ff3(U, X), F  ff3(X, U) is cor,pact and the semigroup generated by A+BF 
exponentially stable. Then there exists -* > 0 such that for every r 6 (0, r*). there 
exist N >_ 1 a,,d u > 0 su.ch that all solutions of (2.3) satisfy ]]x(t)]] _< 
for all x °  X and all t >_ O. 
Proof. Denote the semigroup generated bv A +BF by TBFt). \¥e start by writing 
A as a perturbation of TBÆ(-). By the variation of parameters formula from the 
perturbation theory of semigroups (see, for example, [14], p. 79), we have 
TBF(-)X = T('r)x + T(r - s)BFTBv(s)xds, Vx  X. 
Using (2.10) and defining P  q3(X) by 
Pz:= T(r-s)BF(I-TF(S))xds. Vxe A', 

we see that 
(3.1) A. = TBV(r) + P-. 



STABILITY OF INFINITE-DIMENSIONAL SAMPLED-DATA SYSTEMS 3307 

By the hypothesis that TBF is exponentially stable, there exist AI >_ 1 and w > 0 
such that 
Adopting a standard technique [ronl semigroup theory (see, for exanlple, the proo[ 
of Theorem 5.2 on p. 19 in Pazy [14]), we introduce a new norm I" ] on X by setting 

' 1 
(2 lear y, 
(3.2) 

t>0 

and, lllOl'eover, 
ITF(t)xl 

(3.3) 

Si,me F is compact and lims--.0(I - TBF(8)).r = [) flH" ail x  X, an application of 
Lenmm. 2.1 yields 
(3.4) lira IIF(I - TBF(S))II(X,U ) = O. 
For L e N(X) we denote the operator norm sup,ex([Lz/ml) by L. Using (3.1)- 
(a.a) and invoking the inequality 
e -¢Nl-wre -, VrN+, 

we obtain 

(3.5) 
where 

By (3.4), 

IA,-I < «-' + Ioe,-I < e -' + Mlloe,-II 
<_ 1-wr+[w(1-e-r)+h(r)]r. VrIR+, 

h(r) := M sup IIT(r- s)BF(I l çv(s))ll.(.x-. 
0<s<_r 

lira h(r) = O. 
r--,0 + 
and hence it follows from (3.5) that for fixed but. arbitrary e  (0. w) there exists 
r > 0 such that 
(3.6) I',-I < 1 - (w - e)r < 1, V r  (0, 
Applying (3.2), we see that for any r  (0, 
Combining this wit.h (3.6) shows that A is power stable for ail r 
Remark 3.2. (1) Examining the proof of Theorem 3.1, we see that for every 
(0, w), there exists re > 0 8tlch that 
IIAII  M(1 - (w-e)ç) , Vk  N, Vç  (0, re). 
{2) A key element in the proof of Theorem a.1 is that lim0+ h(r) = 0, which 
implies that 
lira Ilerll _ o. 
r0 + 
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In the case of unl»ounded B, this does hot seem to be true, even if B is an adlnissible 
control operator for T(t) (iii the sense of Weiss [21]). O 
The next renmrk shows that the compactness assumption on F imposed in The- 
orem 3.1 cannot be relaxed. 

Remarie 3.3. It is easy to find counterexmnples which show that, in general, The- 
orem 3.1 is hot truc for non-conlpact feedback operators F. Indeed, consider the 
case whcre X = le(E), A = diagke z(l+ki), 13 = I and F = -2I. Then Fis 
a çontinuous-time exponentially stabilizing feedbaçk, since A +/3F generates the 
expolu,nt ially st al»le selnigroup 
Ts-(t) = diagkez(e(-+Ot ). 
However, applying the sampled-data feedlmck given by (2.3b) results in the discrete- 
tilne opcrator 
l (e (l+ki)r -- 1 ) 
A- = diagkez(e(+O- ) -- 2(liage z \ i 7' " 

Since 

] ( e(+ki)r --1) 2 
c (+ki)'- -- 2 - '4 _> e'- 1), 
Il + iiVl (e + 
we sec that for any r > 0. the eigenvahws ) of  satisfy 
liminf]l 2 e  > 1, 
showing lhat  is hot power stable for any r > 0. 
Furthermore, il is possible to find counterexmnples which show that, in generN, 
Theorem a.1 is hot true when B is not bounded. In particular, in [17] an example 
is given for which U is one-dimensional, F is bounded,  is hot bounded, A + F 
generates an exponentially stable semigroup, A- e  N(U, X) for any g > 0 (so B 
is "barely unbounded" by a colmnon mesure of unboundedness), and there exists 
a sequence of positive sampling times (r) such that r + 0 and  is not power 
stable.  

Remark 3.4. It is important to remise that Theorenl 3.1 is colnpletely general 
in the sense that it applies to any generator A, any bounded control operator 
/3 and. in particular, any compact stabilizing feedback F. Obviously, in specific 
cases, the result could be proved using different, possibly simpler arguments. As an 
obvious example, consider the case of a system with a finite-dimensional unstable 
part (typically arising from a parabolic PDE) and a feedback that stabilizes the 
unstable part and is zero on the stable invariant subspace. More precisely, in this 
case there exists a projection operator 1] on X such that X1 := 1]X (the unstable 
subspace) and X := (I - 1])(X) (t.he stable subspace) are invariaut with respect 
to the underlying semigroup, X is finite-dimensional and X1 C D(A): moreover, 
0 A2 ' B2 ' 
where A := AL-,, A := AIx, /3 := 1]B, /32 := (I - 1])/3 and the operator A_ 
generates an exponentially stable semigroup on X2 (see, or example Lema 2.5.7 
in Curtain and Zwart [3] for more details). I F : X -- U is an exponentialb 
stabilizing [edback or the (finite-dimensional) system (A,/31), then F :: (. 0) 
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is an exponentially stabilizing feedback for (A, B). It is obvious from the well- 
known finite-(limensional version of Theorem 3.1 tiret for this particular feedback 
the con'esponding sampled-data feedback system (2.3) is expolmntially stable for ail 
suffciently snmll r > 0. Clearly, one could imagine proofs tailored to other situations 
with specific feedbacks, say for example hyperbolic PDEs, retarded and neutral 
flmctioual equations or mixed parabolic/hyperbolic systems. However, iii nmnv 
significant applications there is little scope for actually choosing the feedback. For 
example, iii stat.ic output feedback, the basic structure of the feedback is inherited 
from the given observation aud the feedback design is limited to choosing feedback 
gains. Iii such ca,ses it is diflïcult to see the mcrits of specifically tailored proofs 
or indeed how they would be developed. It is therefore an important aspect of 
Theorem 3.1 that it does hot make any assumptions on the SlmCific structure of the 
underlying syst.em (A, B) and the stabilizing feedback F. 
In the next section we show/bat Theorem 3.1 generalises fo the case with un- 
bounded B in the special, but important, case where .4 geuerates an analytic senti- 
group. 

-1. UNBOUNDED CONTIROL AND ANALYTIC SEMIGIROUP 

Iii ortier to fornmlate and prove the main result of this section a nulnber of 
preparations are required. Throughout this section, we aSSUlle that A generates 
a strongly continuous semigroup T(t) on X, B  5(U, X_) and F G 5(X, U). 
Further assuml)tions on (A. B, F) will be introduced «hen needed. % define an 
operator AF on X bv setting 
(4.1) ABFa = (A + BF)«, VxGD(ABF):={,rGX[(A+BF)a'GX}. 
If is easy to show that AF is closed.  carefully distinguish between the operators 
AF and A + BF, the latter being an unbounded operator on X « with dolnain 
D(A + BF) = X. Clearly, A + BF  5(X,X_). Note that AF is the part 
of A + BF in X. The transfer fimction of the contilmous-tine systeln given bv 
(A, B. F) is defined as follows: 
(4.2)  : o(A)  m(u), s  F(- 4)-B. 
 introduce the space X, := D(A*), endowed with t.he norm 
Ilmll. := Il(X«- 
where A is any element of O(A}. Clearly, X  X = X*. Il is well known that X_  
and (X.)* are isometrically isomorphic. Therefore, the triple X,  X = X*  
X-1 is a so-called Gelfand triple. Ioreover, if B  5(U, X_), then B*  (X., lr). 
Lennna 4.1. Under the above assumptions on A. B and F, the followig state- 
ments hold: 
(1) o(A+ BF)£(A)= {s o(A) I 1  o(G(s))}; 
(2) if a(A + BF) ¢ O. then o(AF) = O(A + BF) ad 
(sI- Aer) -t = (sI- A- BF)-]x, Vs60(Ae)=0(A+BF); 
(3) if F is compact, then 
0(A)  o(A) c 0(A + BF); 
(4) ifF is compact, then AS = A* + F'B* with D(AS) = D(A*) = X,. 
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Ploof. To pi'ove statelm,nts (1) and (3), let s E e(A). and write 
(4.3) ,I - A - BF = (si - 4)[1 - (si - A) - BF]. 
It follows that 
s(A+BF)  I ((sI-A)-BF)  I o(F(sI-A)-IB), 
yielding taWment (l). Assume that F is compact. Then, by (4.3), 
,,"  «(A + BF)  I  «p((I - A)-IBF). 
Therefore, if '  «(A+BF), there exists .r X, 
.r. H¢nce, (A + BF)x = s.r, showing that s  «p(,4v). We may conclude that 
a(A + BF)  0(A) C a(Av), proving statelnent (3). 
To prove statement (2), let A  O(A + BF) and note that. by Proposition 2.17 
on p. 261 in [5], we only need to show that the topolo" on X given by the norm 
 ,' lal 
is stronger than the original norm topology. But this follows immediately, since 
with 
# :: Il(A/- 
we bave 
i.e., the two norms are even equivalent. 
To prove statement (4), consider the operator 
D(A*)  X.. The compaetness of F together with the fact that B*  (X., U) 
yields tlmt F'B* is relatively compact with respect 
closed. A* + F'B* is closed (see Theorem 1.1 on p. 194 in [8]) and so 
(A* + F'B*)** = A* + F'B* 
(see Theorem 5.29 on p. 168 in [8]). As a consequence, statement (4) will follow if 
we can show that 
(A 
To thi end assume that   D((A* + F'B*)*). Setting «' := (A" + F*B*)*z X, 
ve bave that 
where (-,-) denotes the inner product in X. The duality pairing on X_ x X., 
denoted by [-,-], i given by 
[U,z] := ((M- A)-Iu.(I - A*)z), V(U,z) 6 X_ x X., 
where  6 (A). It is straightforward to show that 
(4.) 

Thus, by (4.4), 

[x'-(A+BF)x,z]=O, Vz E X. , 
and so ( A + B F ) x = x'  X. proving that x  D ( A B F ) and A B F X = ( A * + F* B* ) * x. 
This shows that ABOe is an extension of (A* + F'B*)* It remains to prove that 
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D(AsF) C D((A* + F'B*)*). Let x E D(AsF). Invoking (4.5), we obtain for all 
zE X., 
(x,(A* + F*B*)) = [(A + BF).%z] = 
showing that x  D((A* + F'B*)*). [] 
Remark 4.2. Statements (2) and (3) of Lemma 4.1 are wrong if the operator ASF 
is replaced by a proper restriction of ABF. More precisely, consider an operator 
sF on X given by 
ÀBFX : (.4 -]- BF)x, Vx  D(ÀBF) C D(ABF). 
If D(ABF) 7  D(ABF), then 
(4.6) O(ÇiB,) C ap(A,,) C ap(A + BF). 
The second inchlsion holds trivially. To prove the first inclusion, let A  
Then, AI - ïBF " D(_ÀBF) --- X is bijective and hence, for y  D(ABF) \ 
there exists x  D(ABF) such that (AI - ÀBF)£ = (AI - ABF)Y. It follows that 
(AI - ABF)(y -- x) = 0, and since y - x =/= 0, we see that A  ap(ABF). 
Moreover, combining statelnent (1) of Lelnma 4.1 and (4.6), we see that if 
is a proper restriction of ABF, then I-- G(s) is not invertible for any s  0(ÀBF) 
o(A). 
Lemma 4.3. Assume that F  B(X,U) is compact and that the following two 
spectral assumptions hold: 
(S1) Ç,o C O(ABF); 
($2) there exists e > 0 such that er(A)  C_« is bounded. 
Then there exists a < 0 such that er(A) ç C« consists of finitely many eigenvalues 
of A with finite algebraic multiplicities. 
Pro@ Since F is compact, it follows from statement (3) of Lemma 4.1 and 
sumption (S1) that o(A + BF)  O. Hence, combining statement (2) of Lemma 
4.1, assumption (S1) and the fact that O(ABF) is open shows that there exists an 
open set ç such that 
(4.7) o(A + BF) = O(ABF) D f/ D Ç0- 
Recall that the spectrum of A considered as an operat.or on X coincides with the 
spectrum of A considered as an operator on X -1 (see [5], p. 261, Proposition 2.17). 
Moreover, the point spectra of A and its extension are identical, including algebraic 
multiplicities of isolated eigenvalues, For the rest of this proof we will consider A 
as an operator on X_ with domain X. Clearly, 
(4.8) sI - A = [I + BF(sI - A - BF)-I](sI - A - BF) , V s  fi, 
and therefore, by the compactness of F, 
(4.9) s  a(A)  -1  ap(BF(sI- A- BF)-), Vs  ri. 
Let s  a(A) f3 f. Then, by (4.8) and (4.9), there exists x  X-l. x  O, such that 
(.si - A)(sI - A - BF)-lx = O, 
showing that s is an eigenvalue of A, and so 
(4.10) a(A) ç ç = ap(A) 1 
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Combining (4.9), (4.10) and the compa«tless of F with Theorem 1.9 on p. 370 in 
Kato [8] shows that for any compact set tç C , the intersection a(A)I" i finite. 
It now fi»llows ffoto assumption ($2) that there exists  < 0 su«h that apA) 
is finite and Ep(A) Ca C . It remains to show that each element in avA ) 
has finite algebraic mnltiplicity. Seeking a contradiction, let A G avA)  Ça C 
and SUpl»OSe that A has iufinite algebraic multiplicit.y. Then. by Theorem 5.28 on p. 
239 in [8], A behmgs to the essential spectrmn of A (as defined on p. 243 in [8]). As 
an uni»omMed perturbation on X_, the operator BF is relatively compact with 
respect to A (considered on X_), due to the compactness of F. By Theorem 5.35 
on p. 214 in [8], the essential spectrmn of an operator remains fixed under relatively 
compact perturlmtions, and thus, A G a(A + BF)  t, contradicting (4.7). 
Assmne that F G (X, U) is compact and the assmnptions (S1) and ($2) hold. 
Then, by Lemma 4.3, there exists « < 0 su«h that a(A)  Ça consists of finitely 
manv eigenvalues of 4 with finite algebrai« multipli«ities. Hence, there exists a 
simple closed curve  in the halgplane Ç llOt intersecting a(A) and enclosing 
OE(,4)  Ç0 in its interior. The operator 
is a projection operator, and we bave 
(4.12) X=XX, where X :=X. X.:=(I-H)X. 
By a standard result (see. for example, Lemma 2.5.7 in Curtain and Zwart [ai), the 
above de«omposition has the following properties: 
(D1) dira X < oe and X C D(A)" 
(D2) X and X are T(Q-invarian for all ¢  0: 
(a) «(AIx) = «(A e0 and «(AIx) = «(A)  (Ç N 
I is uSehll to introdu«e he following notation: 
(4.13) A := A]x,, T(t) := T(t)lx j : 1,2. 
Clearly, by (D1) and (D2), T(t) is a semigroup on the finite-dimensional space X 
with generator A, i.e., Tl(t) = e At, and T(t) is a strongly continuous semigroup 
on X with generator 4.» Since the spe«trum of 4 considered  an operator on X 
coinçides with the spectrum of A considered as an operator on X_, the projection 
operator H on X defined in (4.11) extends to a projection on X_. We will use 
the saine swnbol H for the original projection and its associated extension. The 
decomposition (4.12) induces decompositions ofthe control operator B  (U, X. 1) 
and the feedback operator F  (X, U): 
(4.14) 1 :: HB. B := (I- H)B, Fa :: F[x, F2 := 
Lemma 4.4. Assume that F  N(X, U) is compact and that assumptions (S1) and 
($2) h old. Then the pair (A, B ) is controllable and the pair ( A  , F1) is observable. 
Pro@ Seeking a contradict ion, suppose t hat (A, B1) is hot controllable. Then. by 
the Kahnan controllability decomposition lemma, we may assume, without loss of 
generality, that A1 and B tan be written in the form 
0 A , Bt = 0 " 
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A straightforward calculation combined with an application of statenlent (2) of 
Lemma 4.1 shows that 
(4.16) (sI--ABF)-' = (sI-A)-I+(sI-A)-BF(sI-ABF) -1, Vse 
Using the decomposition of A and B and (4.15), we see that (si - A)-IBF may 

be written in the form 
(4.17) 

(sI-A)-IBF = 0 (si- Al2) -1 0 0 0 0 . 
0 0 (sI-A2) - , , , 
Let A E «(A) and let y be a corresponding eigenvector of A. Note that, by 
(D3), A E Ç0- Setting x := col(0, y, 0) ¢ X, we may conclude from (4.16) and 
(4.17) that 
(si - ABF)-I.r = (SI- A)-y + 0 = (s - A)-y 
0 * * 
This shows that Il(si- ABF)-IXll --- OC as s -- A in 0(A)ç C0, leading to a 
contradiction, since by (S1), A  p(ABF). 
To show the observability of (A, F), note that 
(4.18) 
(si - AUF) -1 = (si -- A) -1 + (si - A - BF)- BF(sI - A) 
which can be obtained from a straightforward calculation combined with an appli- 
cation of statement (2) of Lemnm 4.1. Based on the Kalman observability decom- 
position lemma and (4.18), an argument similar to that establishing controllability 
of (A, B) can be used to prove observability of (A, F). We omit the details for 
the sake of brevity. [] 
The next result shows that if A generates an analytic semigroup and if F is 
compact, then ABF generates an analvtic semigroup. 
Proposition 4.5. Assume that A generates an analytic semigro'up T(t) on X, 
B  (U,X_), and F  (X, U) is compact. Then ABF generates an analytic 
semigro.up TBF(t) on X. Moreover, if TBF(t) is exponentially stable, then (I- 
G) - E H(C0, (U)), where G is given by (4.2). 
Proof. By statement (4) of Lemnla 4.1, Aï F = A* + F'B* with D(A*BF ) = 
D(A*) = X. Clearly, since A generates an analytic semigroup, so does A*. More- 
over, by the assmnptions on B and F, B*  (X, U) and F* is compact, and so 
F'B* is relatively compact with respect to A*. By a well-known result from the 
perturbation theory of analytic senligroups (see Corollary 2.17 on p. 180 in [5] or 
Proposition 1 in Zabczyk [23]), it follows that A*BF = A* + F'B* generates an an- 
alytic semigroup, and hence, so does A* g. By the closedness of ABF, we have that 
A**F = ABF, and thus we may conclude that AF generates an analvtic senfigroup 
TuF(t). 
Assume that TuF(t) is exponentially stable. Then, by Lenuna 4.1, 
and for ail s  Ç0 and for all z  X-1 we have that 
(si- A- BF)-Jx = (A + BF)(sI- A- BF)-I(A + UF)-lx. 
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Since (A + BF)-lx E X, we may conchule, using Lemma 4.1, that for all s 
ail([ ail x 6 X_ 1, 
(sl- 4- BF)-lx = Asv(l- Asv)-I(A + BF)-lx 
(4.19) = (s(sI- ABF) -1- I)(A + BF)-lx. 
A straightforward calculation shows that 
(I-G(s))-I = (I- G(s))-lG(s)+ I = F(sI-A-BF)-IB + I, 
and so, by (4.19), 
(4.20) (I - G(s)) -1 = F(s(sI- ABF) -1 - I)(A + BF)-IB + I. 
Since TBF(t) is an exponentially stable analytic semigroup and since (A+BF)-IB 
N(U, X), the right-hmM side of (4.20) is mfiformlv bounded for s G Ç0- Therefore, 
(1-(I) -1 e H(Co, N(U)). [] 
Corollary 4.6. Assume that A generates an analytic semigroup T(t) on X. B 
N(U, X-l), and F  3(X, U) is compact. If the semigroup generated bg AF 
e.rponentially stable, then there ezists a decomposition X = X @ X.2 such that 
(D1)-(D3) are satisfied and. moreover, the following property holds: 
(D4) T2(t) is au exponentially tablc analytic semigroup on X with generator 
A.2, where T(t) and A.2 a given b (4.13). 
Pro@ By the exponential stability of the scmigroup generated by 4F, assumption 
(S1) in Lcnmm 4.3 holds. Since A gencrates an analvtic semigroup, for an5"/3  11, 
the intersecti(m «(A)ç)C is bounded, and so assmnption ($2) in Lemma 4.3 
is satisfied. Therefore, by Lemma 4.3, properties (D1)-(D3) hold. Moreover. it 
is clear that the semigroup T2(t) is analytic, and therefore satisfies the spectrum 
determined growth assumption. By (D3) and Lemma 4.3, the generator A._ of 
satisfies «(A2) C C \ C-e for some e > 0. Therefore, we may conclude that Tu(t) is 
exponentially stable. [] 
Consider the discrete-time system (T(r). Sr, F), where Sr is given by (2.6), and 
denote the transfer function of this system by Hr, i.e., 
(4.21) H-(z) = F(zl- T(r))-lSr. 
Then, under the assumptions of Lcmma 4.3 (i.e., F is compact and ($1) and ($2) 
hold), we may write 
(4.22) n = H + H, 
where, using the notation of (4.13) and (4.14), 
(4.23) H(z) := Fj(zI- ri(r)) -1 Tj(s)Bds. j = 1.2. 
In light of Lelnma 2.3, in order to prove that the sampled-data system (2.3) is 
exponentially stable for a given sampling time r > 0. we need to show that 
given by (2.10) is power stable. The following lemma provides a sutïficient condition 
for the power stability of A_ in terres of r and H. 
Lemma 4.7. Assume that ,4 generates an analytic semi9roup T(t) on X, B 
(U, X _). F  (X, U) is compact and the semigroup generated by ABF is expo- 
nentially stable. If r > 0 is such that 
(4.24) r(A-Al)¢2krci, Vk e Z\{0}, 
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and if (I - H,-) -1 E Ha(E, (U)), then Af is power stable. 
Note that under the assulnptions of the above lemma, (4.24) is satisfied for all 
sufficiently Slnall ï > 0. 

Proof of Lemma 4.7. By Corollary 4.6, there exists a decolnposition X = A'I @ X2 
such that (D1)-(D4) hold. In particular, using the notation of (4.13) and (4.14), 
we have «(A) = «(A)Ç0, and, lnoreover, setting S := fO Tj(s)Ba ds, j = 1,2, 
we mav write 
0 ()  = s '  
Assulne that  satisfies (4.24). By Lclnma 4.4. the finite-dinmnsiollal controlled 
system (A, B) is controllable, and so, by a well-known result (see [7] or Theorem 4 
on p. 102 in Sontag [19]), the discrete-time systelu (T (), S) is controllable as well. 
This implies, iu particular, that Ihel-e exists Iç E (X, U) such that the lnatrix 
Tl(7) + SIÇ 1 i powPr [ablo. No- (T) i8 i)owr s[ale (illce [lle 8çllligroilp (t) 
is exponentially stable by (D)) and, setting Iç := (IQ, 0) E (X, U), it follows 
frolu (4.25) that T() + SIç is power stable, showing that the controlled discrete- 
tilne systcm (T(), S) is stalfilizable. A siluilar argmneut shovs that the observed 
discrete-time system (T(), F) is detectable. Hence, if (I - H) -1 E H(E, (U)), 
we mav conclude that A is power stable (see Theorem 2 in Logemalm [9]).  
The following theorem is the nlain result of this section. 

Theorem 4.8. Assume that ,4 generates an analytic semigroup T(t) on X, B  
(U,X_I). and F  (X,U) zs compact. If the semigroup generated by ABF iS 
exponentially stable, tben there exists ï* > 0 such that for every ï  (0, -*), there 
exist N >_ 1 and  > 0 such that all solutions of (2.3) satisfy IIx(t)ll < Ne-tl]x°]] 
for all x °  X and all t >_ O. 

We see that the above result is of the saine form as Theorem 3.1. In particular, 
no assulnptions are ruade on the specific structure of the stabilizing contilmous-tilne 
feedback F. It is therefore clear that Relnark 3.4 also applies to Theoreln 4.8. 
The following lemlna is the key tool for thc proof of Theorem 4.8. 
Lemma 4.9. Forç > O, define I(ç) := {z=e   sGC0,sl < ç}. Underthe 
assumptions of Theorem 4.8, there exist ç > 0 and 0 > 0 such that 
(4.26) (I- H) - H(IV(ç),(U)) Vr(O,O), 
where H is given by (4.21). 
 defer the long and techlfical proof of Lemma 4.9 to the end of this section. 
Proof of Theorem .8. By Lelmna 2.3 and Lemma 4.7, it is sufficient to show that 
there exist.s * > 0 such that 
(4.27) (I- H) -1  HOe(E,(U)), V  (O,T*). 
To this end, let M  1 and    be such that 
(4.28) IIT(t)ll  AIe', Vt  +. 
We split E into three disjoint parts: 
(4.29) E = E(5) U E(5) U E3, 



3316 H. LOGEMANN, R. REBARBER, AND S. TOWNLEY 

where, for 5 E {0, 1), 
and E3 := {z  C llzl 2 4M}. Ve proçeed in several steps. 
Step 1:  claire that there exist 5 > 0 and ç > 0 such that 
(4.30) (-H) - eH((6),(U)), Ve (0,). 
By Lemma 4.9 there exist / > 0 and 0 > 0 such that (4.26) holds. Choosing 
 E (0, 1) such that El(5) C IV(), we sec that (4.30) follows with 7 = 0. 
Step 2: Let  E (0, 1) be as in Step 1.  claire that there exists T2 > 0 such 
that 
(4.31) (I-H)-' (E2(6),(U)), V (0,:). 
To this end, first note that by (4.23) 
H(z) = F((z- 1)I-(T(r)- I))- T(s)B ds. 
Sinco T (¢) is a Sonligroup on lle finite-dinlensional space A', we have 
lira I[T(ç)- Ill = O. 

Moreover, 

0 "V 
lin, 'I rl()B, d, ll = 0. 
r---0 
Since Iz - 11 > 6 > 0 for all z E E2(6), a routine argument involving the application 
of the Neumann series shows that there there exists 0* > 0 such that 
(4.32) IIH(z))ll < 114, Vz e E(), Vr e (0,0"). 
Now we analyze H(z) on E(), the closure of E2(). Using analyticity and 
boundedness of the senfigroup T2(t) (guaranteed by (D4)), an application of The 
rem 5.6 (b) on p. 66 in Pazy [14] shows that for each A e E() there exist 0x > 0 
and kx > 0 such that 
(4.33) Il(A/- T(r))-II  kx, VE e (0.0x). 
For z, A e E(6) we have 
Il(z/- T2(ç))-I l[  II(T(r) - al)-llll(l- (z- a)(T(r) - aI)-')-'ll - 
Combining this with (4.33), an application of the Neumann series yields that for 
each A  E(), 
I1(I- ())-11 s ex, vz e (a, /(ex)), v e (0.0x). 
Since E(6) is compact, we may conclude from a standard compactness argument 
that there exist k > 0 and  > 0 such that 
(4.3a) I1( »- ())-'11S , V e E(6),  e (0,). 
H(z) = F2(T2(ç) - l)(zI - T2(ç))-]AI B2, 
and since F2 is compact, it follows from Lemma 2.1 and (4.34) that there exists 
Ô  (0. ) such that 
IlU()ll s /4, vz e £(),  e (0,ô). 
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Colnbining this »vith (4.32) and (4.22) sho»vs that 
IIHT(z)ll _< 1/2, 'dz e E2(5), 7- e (0, r2), 
where ç := lnill(0*, 0). 5t another application of the Neumalm series yields (4.31). 
Step 3: We claire that there exists ça > 0 such that 
(4.35) (- H) 
By (2.10) and (4.28). together wih Lemma 2.2, here exists va > 0 such that 
I,ll  2. v e (o,3). 

Defining the open set 

(4.39) 
wh'I'' 
(4.40) 
Using 
(4.41) 

Ê3 := 
a routine argmnent iuvolving the Neummm series then shows that 
(4.36) II(:Z-)-Xll=lz-llll( z-z-)-lll l/M, VzeÊ3, Vçe(O, ç3). 
Now 
(4.37) (I - H(z)) - = I + F(:I - A)-S, 
and so (4.35) follows ri'oin (4.36), (4.37) and Lemma 2.2. 
Step 4: Finally, setting r* = lllill(T 1, T2, T3), a COlllbillatiol of (4.29)-(4.31) and 
(4.35) yields (4.27). 
It relnains to prove the crucial Lemma 4.9. 
Proof of Lemma 4.9. Recall the definition of G in (4.2). Appealing to (D1)-(D4), 
(4.13) and (4.14), we may write 
(4.38) G = Gt + G2, where G)(s) = Fj(sI- A,) -Bj, j = 1,2. 
It is convenient to set 
Cç := C0  «(A) = C0  «(Al)  
For l > 0 and 0 > 0, we define 
The idea of the proof is to compare H(e ¢) to G(s) for ail (ç, s)  V*(0. ), where 
 > 0 and 0 > 0 are sufficiently slnall.  proceed in several steps. 
Step 1: In this step we analyze the terre H(e ¢) - G(s).  claire that 
Gl(s)- H(e 
Q is a matrix-valued fimction defined on (0, ) x C such that 
li 
(0,) 0 
the finite-dimensionality of X, we obtain using Taylor expansions 

where 

Tl(t)/31 dt = r(I + QI(r))/31 , 

- «-TSTl(r)= T( + Q_(T,s))(sI- Al), 

oc Tk_ 1 oc Tk_ 1 
QI(r) :: Z k--Ç-. Aî-l" Q2('r"s)1= Z k-. (Al - "si)k--l" 
k-=2 k=2 
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Notc that Q1 and Q2 satisfv 
(4.42) lira Ql('r) = 0, lira Q2('r, s) = 0. 
TO (T,S)-O 
The first limit follows immcdiatcly flore thc dcfinit ion of Q, whilst the second limit 
follows from the inequality 
I[Q(,*)[I  ('+)- 1, 
which is t he result of a rout inc cs imat c. Using (4.41 ), a st raight forward calculat ion 
shows that (4.39) holds wih 
(4.43) Q(ç,s) := r[(1 -e-')I-e-Q(r)+Q2(r,s)](M - «,)(I-e-T(r)) - 
Note that. i W thc spectral mal)l)ing thcorem. I- e-ZST(r) is invertible for all 
(v, s)  (0, )  C, and so. Q(r, s) is w(,ll-defined for ail (r. s)  (0. )  C. From 
(4.41) we obtain 
Q(r, s)= [(1-e-)l-¢.-Q(ç)+Q2(r,s)](I + Q2(r, s)) - , 
and it fi)lh)ws from (4.42) that Q satisfics (4.41)). 
Stop 2: k show that 
In or(let to estimate the differonce Hç(e s) - G.(s), note that 
(«) - G() =  [(«T - T())-(V() - T)A]' - ( - A) -]  
= 2 [(T2(r) - I)A(sI -A) - (eI - 
x (eT - T(v))-(T - A)-B. 
Setting B2(s) := (si - 42) -, routine algebraic manipulations show that 

(4.45) 
where 

1 - e -s 
I 
T8 
converges strongly to 0 as ('r, "fs) - 0. the compactness of F2 together with Lemma 
2.1 shows that 
(4.46) lira [[Pl('r,s)]]----O. 
(r,-rs)-0 
Estirnating P.2('r, s) is considerably more difficult. In view of (4.45) and (4.46), to 
establish (4.44), it is suf5cient to prove the following claire. 
Claim: For any p  (0, zr) there exist  > 0 and k > 0 such that 
(4.47) [[P.(-,s)[[ _< k < oc, V(T,S) 6 V(,p). 
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Pvoof. Since the semigroup T2(t) is exponentially stable, for fixed s E Ç0 and 7- > 0 
the spectral radius of e-TST2(7.) is smaller than 1, and the Neumann series gives 

(4.48) (I - e-TST2(7.)) -1 = E e-kTT2(kr) " 

Since T2(t) is an exponentially stable analytic semigroup, there exist/3 E (0, rr/2), 
w > 0 and 1I > 0 such that 

aild 
(4.49) 

o(A2) ::3 E := {ç e C I larg( c + w)l < rr/2 +/3} 

IIn2(OII < I/11, v  = \ {o}. 
If we note that R2(ç)B2 = (R2() - I)AIB2, (4.49) shows that for any E > 0 
there exists N > 0 such that 
(4.50) IlU2(ç)B211w,x)  N, V e . 
Let   (/2,/2 + ff) and Vo  (O,w). Define the COl,tour 
r = {-* 1 > 0} u {0 * [ 4 e [,2 - 1}  {d 1 > 0}, 
oriented bottom to top. Note that by construction F C E and a(A) is to the left 
of F; see Figure 4.1. 

F 

FIGURE 4.1. The contour F with «(A2) to the lefl of F 

Invoking (4.48) and using a standard relationship between T2(t) and R2(s) (see 
[14], Theorem 7.7, p. 30), we have 

(4.51) 

(I e-ST . -1 1 J r 
- 2(7.)) = I +  e-eT'R2(A) dA. 

By (4.49), 
II-%k(A)ll <_ ] l, 

VA  F, Vs  Co, VT. > 0. 
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Hence we ca.n use Lebesgue's (lominated convergence theorem to interchange the 
order of Sllllllllat Joli raid int egrat Joli iii (4., 1). Evahmt illg t he result ing infinit e sure. 

we oi)t ain 
(4.52) 
Setting 

I [ rse T(-») R2(A)R.(s)B2 

(4.53) jï(T, A, s) .- _ «(»_) _ , 
1-e(-) + = -1 
and letting p  (0, n) i)e fixed, hut arbitrary, an elcmcntarv analysis (sec the Ap- 
pendix) shows that 
(4.54) 
,,,, {l'' I I (, .) (o,,)x r x Co_ s.t. rlAlp,lslp}<. 
Definc  := p/r 0 and for r  (0, ) introduce 
r. := ( e r 1  p}. 
Clearly, I'r ¢  for all r G (I), ).  (lecompose the integral on the right-hand side 
of (4.52) as follows: 
 rser(-s ) 
(4.55) I(,) := i)()()=,(.)+I(.)(), 
where 
I(r,s):= . l_e.(_)R(A)R(s)BdA, /(r,s):= kr. 1-e'(-)R(A)dA" 
% first analyze I(r, s). Using the resolvent identity 
(- )n()n()= n()- n(), 
and invoking (4.53), we obtain 
(4.6) I(r,s) = f(r,A.s) R(A) R(s) BdA + 
Setting 
[ll(T,$) := /(T,X,$) 2(x) R(s)BdX I,(r,s) := 
. s-A ' .s-A " 
we bave 
Invoking (4.50) and (4.54) shows that there exists k > 0 such that for all (r. s) G 

II&l(r, s)ll 

v(,p), 

ff If(,A.s)l 
< . lA- si (lln(A)ll + IIn(s)ll) dlAI 
_< kr length(F-). 

Conlbining this with 
and setting/,:2 := 2k(7r + 1)p yields 
(.58) I1,, (, )11 _< 

length (F.) _< 2(rr + 1)p/r 

v (r, s)  v(& p). 
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To estimate 112(% s), we define 

and for each r  (0, 5) we embed F- into a closed contour 
c,- := {ï-" I  e [,()1} u {e  I ¢ e [, - ]} 
u{  I e [, oCé)I} u {Æ()'¢ I ¢ e [-, ]}, 
oriented clockwise (see Figure 4.2). Then 
Jc s 
I2(ç, s) = (s AR(A) dA - R2(rei)e i dr 
 v/ s- rei 
f() s J .is()e ¢ 
(4.59) + 
/ s- re 
By construction, for a given ç  (0, 5), any s  C  (0.p/) is inskle the contour 
C and so Cauchy's integral fornmla combined with (4.49) shows that there exists 
ka > 0 such that 
(4.60) I[Jc, (s s xR(x) dXll = 2nllsR(s)ll < ka_ , V(7, s)  V(5,p) . 
Moreover, since I s -re'e I > r[cos@[ for s  C0, it follows from (4.49) that there 

exists k4 > 0 such that 
]Jp/r s - re i R2(rei)eidr 

As an immediate consequence we see that 

(4.61) 

v(r,s)  v(5,p). 

s-rei 

_ k5, 

v(r,s)  v(5,p), 

FIGURE 4.2. The contour F- embedded in a closed contour C- 
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for some suitable/,'5 > 0. Similarly, there exists k6 > 0 such that 
(4.62) Jp/7- s - re -i'p - ' " 
Trivially, since gT) = 2p/T, 
I-.()«1 > pi'r. v(,)  v(a,»), v¢ [-¢,¢1, 
and hence, using (4.49), we see tha.t there exists k7 > 0 such that 

(4.6:5) < 2/,'fO, V (r, s)  I'(& p). 

Combiniug (4.59)-(4.63) shows that there exists ku 
(4.64) I1,(,))11 _< , v(.)  
By (4.50) we mav coiclu(lc that Ii2(T, s)R2(s)B2 is 
a.l(1 therefl)re, l)y (4.57) and (4.58), we ol)tain that 
(4.5) ll&(,))ll  »9, v(,)  
To estimate I2(ç, s), first noie that there exists km 

> 0 such lhat 
I'(a.»). 
uniformly bounded on 
there exists k0 > 0 such that 
> 0 such that 

Il - e'('-*)l > klO > 0. V (r, A, s) G (0, oo) x F x Ço s.t. A G F \ Fr. 
Cmbining this wih an application of (4.49), we obtaiu 
(4.66) 11*2(r, )11 < 2,,p f ««°, 
-- -- dr, V (r, s)  V(& p) 
- km /, r ' 
for some suitable constant k > 0. Since cosO < 0, 
 erV cos  ç ep cos  < < 
dr < e p coe  
Together with (4.66) this shows that there exists k2 > 0 such that 
(4.67) ll&(¢,.)ll  1, V(¢,)  V(,p). 
Çombining (4.52), (4.55), (4.65) and (4.67) and invoking (4.50) shows that (4.47) 
holds, completing the proof of the claire.  
Step 3: By sumption, the analytic senfigroup generated by ABF is exponen- 
tially stable. Therefore, by Proposition 4.5, 
(4.68) (1- G) -1  H(C0,(U)), 
which, in particular, implies that I - G(s) is invertible for all s  Ç. Setting 
D(¢,) := G()- H(«'), V(¢,)  (0,)  C0, 

(4.69) 
and 
(4.70) 

Q(¢,):= (Q(¢,)0) . 
0 0 3(X_,X), V(r,s)(0, oc) XCo, 
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with Q given by (4.43), we obtain from (4.38) and (4.39) for all (-, s) 
t hat 
I-H,-(e")) = I-G,(s)-Ge(s)+F(sI-A,)-IQ(-,s)B+D(-,s) 
= (I-G(s))[I+(I--G(s))-'F(sI-A)-'Q(s,r)B 
(4.71) + (I - G(s))-D(r,s)]. 
A straightforward calculation combined with an application of statements (2) and 
(3) of Lemma 4.1 shows that 
(t - ())-F(I - .4)-  = F(t - A - 
(4.72) = F(sI- AF)-z, Vs  C. V  X. 
Moreover, by (4.70), 
(4.73) (Q(,)B)u c x, v(,) e (0,) x c;. 
Combilfing (4.71)-(4.73) shows that for all (, s) G (0. oe)  Ç, 
(4.74) t--H(« ) = (I- ())[t+F(.,t-.4«)-'ç(,.)B+(t-())-D(,)l  
By assumption, the analytic semigroup geuerated by ABF is exponentially stable. 
Çonsequently, the flmction s  (si- .4BF) - is in H(Ço,(X)), and thus we 
may conclude ff'oto (4.40) a.nd (4.70) that 
(4.5) lira (,,p{lIF(.t- A«)-'ç(,.)BII I (.) e '*(0.)})=0. 
(0.n)0 
om (4.44) and (4.69) combined with (4.68), we obtain 
(4.76) lin (sup{](I-G(s))-D(r,s)]]] ] (r,s)  1(0,)})=0. 
(o,) o 
Finally, combining (4.74)-(4.76) shows that that there exist 0 > 0 and  > 0 such 
that (4.26) holds. 

5. EXAMPLES 
In this section we give examples which illustrat.e Theorem 4.8. We consider 
a gelmral parabolic partial differential equation with Dirichlet control. For two 
different types of stabilizing feedbacks u = Fx round in the literature on control of 
partial differential equations, we show that the associated sampled-data feedback 
(1.2) stabilizes the system for small enough - > 0. 
Let ç be a convex bounded open set in I ', n k 2, with C  boundary F. Let 
L(ç,O) = y a(«)O ° 
with smooth real coefficients a a, where 0 is the spatial derivative operator. %re 
consider the following parabolic system: 
0x 
(5.t) -ff{(t,)=-L(,O)a(t,)in(O, oo)×a, x(t,ç)=(t,ç)in(O, oo)×F. 
An example of such a system is a heat equation in a disk. 
Let X = L2(ç), U = L2(F), and let A be the operator -L({,cg) with domain 
©(A) = {x E H2(ç) [ x]r = 0}. It is well known that A is the generator of an 
malytic semigroup; see for instance [4]. Define the Dirichlet map D by 
x = Dy if L({,cg)x = 0 in fi and x : y on F. 
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Feedl)ack stal)ilization of (5.1) is studied in Lasiecka and Triggiani [10], [11], [12]. In 
[11] it is shown that (5.1) tan be written in the forln k = Ax+ Bu with B = -AD. 
Frein standard elliptic theory, D E (L-(F),H1/-()) (see for instalce [13]); se we 
see that B E (U, X-l). 
\Ve new describe two examples of colnpact feedbacks that exponentially stabilize 
(5.1). 
Example 1. We first considcr the tank m feedback frein [10] of the ferre 
m 
F.T(t, .)= 
j=l 
where w-  L-(çt) and 9  L(F)  Thcorem 1.2 frein [10] gives conditions on 
and gj that guarantce that A + BF gencrates an exponentially stable semigroup. 
Sincc F is ol)viously comI)aCt, wc can apply Thcorem 4.8 te conclude that (1.2) 
exl)(mentiall.v stabilizes (5.1) for small cnough - > 0. 
Example 2. Hcrc thc stabilizing fecdback arises frein the solution of a linear 
qu(lratic optimal control I)robh'm. Consi(ler thc test fimctioual 
 (..) (11(  
where x is the solution of (5.1) corresponding te the control function u. It is shown 
in [11] that the optimal control is given by the feedback 
0 
(5.2) u - OuA. Px, 
where O/OA* is the ce-normal dcrivative with respect te A* and P 
is a nonnegative selfadjoint operator which is the unique solution of the algebraic 
Riccati equation 
(x,y}L() + (Px,,4y}L() + (Ax. Py}L() = < O-4. Px, o-A. PY) L(r) 
It is shown in Theorem 2.8 in [11] that the control (5.2) can be written as u = Fx, 
where F  $(X, U). Furthernlore, it is shown in the proof of Theorem 2.10 in [11] 
that BF cau be written in the form AK, where K E $(X) is compact. Hence 
we can apply Theorem 4.8 to  = Ax + Av with feedback v = Kx, to conclude 
that (1.2) exponentially stabilizes (5.1) for small enough - > 0. In [12], Galerkin 
approximations of the feedback (5.2} were developed and shown to be exponentially 
stabilizing for suflïciently small mesh paranleters. These Galerkin approximations 
produce finite-rank feedback, and therefore Theorem 4.8 can again be applied. 

6. DYNAMIC SAMPLED-DATA FEEDBACK 
In this section we apply the results of the previous sections to dynamic sampled- 
data feedback. For simplicity and the sake of brevity, we restrict ourselves here 
to systems with bounded control and bounded observation. The svsteln fo be 
controlled, the plant., is formally given by 
(6.la) 2(t) = Ax(t)+Bu(t), t>O: x(O)=x °X, 
(6.lb) g(t) = Cx(t), 
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where A  D(A) C X --, X generates a strongly continuous semigroup T(t), B E 
(U, X), C Ç (X, }'), and X, U and }" are Hilbert spaces. 
Consider a com.inuous-time controller of the form 
(6.2a) (t) = A«z(t) +Boy(t), t  0; z(0) = z °  
(6.2b) w(t) = Cz(t) + Dçv(t), 
where A«  D(A«) C X«  Xç generates a strongly continuous semigroup Te(t), 
B«  ( X«), C«  (X«, U), D«  (]ç U) and X« is a Hilbert space. Using the 
output (6.lb) of (6.1) as the input for (6.2) and the output (6.2b) of (6.2)  the 
input for (6.1), i.e., 
(6.3) v=g and u=w, 
we obtain the feedback interconnection of (6.1) and (6.2). It is convenient to define 
A 0 D:= . Ç:= , 
":= 0 .4 " 0 B 0 C« Iy 0 " 
Clem'ly, . generates the strongly «ontimous semigroup 
(t) := 0 Te(t) " 
 say that the cont.inuous-time feedback system given by (6.1)-(6.3) is exponen- 
tially stable if the strongly continuous semigroup generated by 
is exponentially stable. 
Let T > 0 and consider the following discretization of (6.2), which is obtained 
by applying the standard hold and salnpling operations to (6.2): 
z 0 
(6.4b) w = C:z + 
We use the discrete-time svstem (6.4) to control the continuous-time system (6.1) 
by sampled-data feedback, i.e., we consider the feedback interconnection of (6.1) 
and (6.4) given by 
Vk = y(kT) and U(kT + t) = Wk , t  [0, T); kN0. 
Via an application of the variation of parameters formula to (6.la), this leads t.o 
the following sampled-data feedback equations: 

(6.5a) 
x(kT + t) = T(t)x(k-) + --./t 
(6.5b) 

T(s)B(D«Cx(kT)+C¢zk)ds, t G [0, T); X(0)= x °, 

0 T 
Zk+l : Tc(T)Zk q- Tc(s)BcCx(kT) ds, k  1% zo = z °. 
The sampled-data feedback system (6.5) is called exponentially stable if there exist 
N > 1 and v > 0 such that for all initial conditions (x °, z °)  X × Xç, 
II(:(#,- + t).z,)ll _< Ne-'«'I+)II(«°, :-°)11. Vt  [0, -), V E 1o. 
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By a. straightforward calculation it follows fronl (6.4a) that 
Ç x((/'÷l)'r) ) :'7r Ç x(/<'r) )Zk+l Zk ' 

where 

fO T 
A-x x x - x x xÇ, }  f()}+ (s)Bh'Cc 
The proof of the following h'mlna is a routine exercise and is therefore left to the 
reader. 

Lenmm 6.1. For any r > O. tbe sarnpled-data feedback system (6.5) is exponen- 
tially stablc if and only if ÀT is power stoble. 
\Ve are now in position to fornmlate the main result of this section. 
Theorenl 6.2. Assume that the operator C is compact. If the cotiuous-time 
feedback system (6.1)-(6.3) is exponentially stable (i.e., the strongly continuous 
semigroup geerated bg ,il + 1C is exponetially stable), then there exists r* > 0 
such that for every r  (0, r*) the sampled-data feedback system (6.5) is exponen- 
tially stable. 
Proof. Introducing the operator 
- - U x ," --, x x x,   (s)[ ds, 
we may write A_ = J(T) + ç/C, and we see that  is of the form (2.10). By 
the compactness of KC and the exponential stability of the senfigroup generated 
by/ +//'C, it follows from Theorem 3.1 and Lenmm 2.3 that there exists -* > 0 
such that for every "r ¢ (0, "r*) the operator ,k_ is power stable. An application of 
Lemma 6.1 yields the claire. [] 
Trivially, the compactness assumption on/'C is satisfied if the observation op- 
erators C and Cc are compact, hl particular, conlpactness of/-C is guaranteed if 
U and Y are finite-dimensional. 
Under suitable assumptions, Theorem 4.8 tan be used to extend Theorem 6.2 to 
systems and controllers with unbomlded control operators B and Bc, respectively, 
provided the senfigroups T(t) and T(t) are analytic. For the sake of brevity we 
omit the details. 

APPENDIX: DERIVATION OF (4.54) 
Let p ¢ (0, 7 0 and consider the meromorphic function g defined on C by 
l+ç-e ç 
(ç) - ç(e«- ) " 
Çlearly, for any «  (0, 2p), the flmction g is bounded on (0, 2p) \ (0, «). More- 
over, invoking the Taylor expansion of e ç or using L'H6pital's rule, we have 
lira g(ç) : -1/2. 
Therefore, we lnay conclude that 
(A.) P{l(ç)l I ç  (0, 2p), ç ¢ O} < . 
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By a straightforward calculation it follows from (4.53) that 

Noting that 

ly(T, , s)l Isl 

Ig(T(- ))1. 

sup{ll/l - Ail (,) e r × c0} < 2, 

we see frolll (A.I) that 

Il (,,) e (0,,) x r x Co 

which is (4.54). 

s.t.. "r[A} _< p, TII <_ p} < 
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APPROXIMATIONS FOR GABOR AND WAVELET FRAMES 

DEGUANG HAN 

ABSTRACT. Let b be a franle vector under the action of a collection of tlili- 
tary operators/4. Motivated by the recent work of Frank, Paulsen and Tiballi 
and some application aspects of Gabor and wavelet frames, we consider the 
existence and uniqucness of thc best approximation by normalized tight frame 
vectors. ¥e prove that for any frame induced by a projective unitary represen- 
tation for a countable discrete group, the best normalized tight frame (NTF) 
approximation exists and is unique. Therefore it applies to Gabor frames (in- 
cluding Gabor frames for subspaces) and frmnes induced by traaslation groups. 
Similar results hold for seni-orthogonal wavelet frames. 

1. INTRODUCTION 
A franle for a separable Hilbcrt space  is a sequence {xn } of 7- such that there 
exist A, B > 0 with the property that 
holds for all x  . The optimal constants (nmxinml for A and minimal for B) are 
called frame bounds. When .4 = B = 1, {x, } is called a normalized frame. Fralnes 
are generalizatious of Riesz bases in the sense that a frame allows linear dependence 
among its elemeilts. This redundancy property can lead to more freedom wheu 
constructing atoms, i.e., frame elements for specific types of expansions {cf. [BHW], 
[BT], [Dan]). From the geometric point of view, a frame for a Hilbert space  is a 
compression {under an orthogonal projection) of a Riesz basis for a larger Hilbert 
space/, while normalized tight frames are compressions of orthonornlal bases (cf. 
[nL]). 
Let {x,} be a frame for . Then S defined bv 
is a positive invertible bounded linear operator on 7-/, which is called the frame 
operator for {x}. A direct calculation yields 
X = Z(x,S-I/2xn>S-I/2xn = Z(x,S-lxn>Xn X  -, 
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which implies that {S-1/2,r,} is a normalized tight frame. The franle {S-l:rn} is 
ç'alled the çaoni«al (or sta,,«ard) du.ai of {x}. h, general, a frmue {g} is called 
a dual fin {x.} if 
ho]ds for everv a" G H. Two framcs {a'n} and {y,,} for H are said to be sim.ilar if 
thcrc is a boumtcd invcrt.ii)]e operator T on H such that Tx, = 9 for ail n. If two 
norma]ized tight fialues are simi]a.r bv an invertii)]e operator T. then T must be al 
unitarv operatur [HL]. 
Motivat«d i)y t]m Grmu-Sc]tmidt pro('ess for a ]iuea] independen set {], ..., wm} 
Pau]scn and Tii)alli in [FPT] investigated t he s9mmetric approzimçtios of frames 
i)y norma]ized t.ight ones. Let {.r,} be a frame for H. A normalized tight frame 
{E,,} for H is salid to })e al sgmmetriç tzpprozimation of {.r,} if it is si,ilar to {xn} 
and tire inequa]it.y 
is valid for all normalized tight frames {z} of II that are similar fo {x}. One 
of the main results in [FPT] states that the symmetric approximation for a frame 
Another approximation was introduced by Balan in [Bai by using the closeness 
bon. A fran,c {,,} i sid t bc«to« ¢o a fra,ne {x.} if thc,'e is a mm,ber   0 
such tha 
for ail c = {c} G /2(N). The infimum of such A is called the closeness bound of 
the frame {y,} to the frame {x.} and is denoted bv c{x.}, {y,}). The coseness 
relation is hot an equivalence relation since it is hot reflexive in general. R. Balan 
defined t he dist ance, db( { x }, { y, } ), between { y } and { x, } by log(d ° ( { xn }. { y }), 
where 
It turns out that (see Theorem 2.4 in [Bai) db({x}, {y}) < oe if and onlv if {x} 
and {y} are similar. In this case we in fact bave 
where Q is the invertible operator that induces the similarity between the two 
fra.mes. Therefore the distance between inequivalent frames is a]ways infinity. Balan 
is also able to compute the mininml distance between a given frame and the tight 
frames. 
In applications the most important and practical frames are the ones that are 
generated by a single vector in a Hilbert space under the action of a suitable coll- 
tion of unitary operators. Vavelet frames and Gabor frames are typical examples. 
A unitary system H is a countable set of unitary operators acting on a separable 
Hilbert space  that contains the identity operator. XX say that a vector ¢ ¢  
is a coplete frame vector (resp. complete noralized tight frame vector) for H 
if HO :: {U¢ " U ¢ H} is a frame (resp. normalized tight frame) for . When 
HO is an orthonormal basis for , ¢ is called a complete wandering vector for 
H. We remark that all these frames are special uiform frames (a uniform frame 
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is a fralne with the property that ail the fralne elements have the saine norln). 
For an extensive discussion of uniform fralnes (especially Ulfiform tight fralnes in 
finite-dimensiolml spaces) we refer to the recent work [BF] and 
Two fralne vectors  and 1 for a unitary systeln  are said to be sivilar (denoted 
by @  ¢1) when the two franles lg and l are silnilar. Write ç(U) for the set of 
ail the COlnplete norlnalized tight fralne vectors for U. Given a ffame vector  for 
a unit.ary system , it is natural to ask which norlnalized tight fralne vector is the 
closest to . XVe not.e that in this setting neither the svlnmetric approximation nor 
the Balan's distance makes scnse for this lroblem since (a) for anv two different 
vectors { and ,1, uu ]]U( - U,I[[ 2 = oe if  is gll infiniLe set and (b) { and 
are not silnilar in general. ç say that {  ç(U) is a best normalized tight frame 
(NTF) appromation for a given COlnplete fralne vector  for  if 
I1« - 11 = dist(«, ç(U)) := inf{]],l- g,][: W  W(/g)}. 
For a given unitary svsteln  and a complete fralne vector , we lnight expect 
that there always exists a best NTF approximation for g). However, this is false, 
since there exist, unitary systmns g4 that adlnit complote Daine vectors but do hot 
adroit any normalized tight frame vector. (For instance, let I r be the Ulfitary 
opcrator on 2 that is the rotation opcrator by =/4, and let  = {I, U}. Then 
adlnits colnplete ffame vectors but does hot adlnit tight ones. One can easily refine 
this to the infinite-dilnelsional complex Hilbert space case.) Therefore it is natural 
to ask the following: 
Question 1. Suppose that a unitary system  adlnits a complete norlnalized tight 
fralne vector. Whcn do we have a best NTF approximation for each complete ffame 
vector? Is the NTF approxilnation alwws unique? 
In the case of Gabor Dames this question h been addressed previously by 
Daubechies, Frank, Feichtinger, Paulsen and Tiballi, etc. The purpose of this paper 
is to answer this question for some important frames including Gabor fralnes and 
wavelet frames. 
X recall solne definitions and notation about Gabor and wavelet Dames. Let 
A be a flfil-rank lat.tice in d x d, and let g(a)  L2(d). The Cahot famil 
associated with A and g is the collection: 
Such a family was first introduced by Gabor [Gai in 1946 for the purpose of signal 
processing and is still both theoretically appealing and successflflly used in appli- 
cations. For some recent developlnents we refer to the book IFS] by Feichtinger 
and Strohmer, and a survey paper [Ca2] by Casazza. When Ç(A.g) is a frame for 
L2(d), we call g a Gabor frame generator. 
Let I be a real expansive matrix (i.e., ail the eigenvahles of I are requir 
to bave absolute values greater than 1). An (kl-dilation) wavelet frame (resp. 
normalized tight "wavelet frame) is a single function   L( d) with the propertv 
that 
m 
¢',e(x):={}detkl[O(kl x-f)" mZ, fZ } 
is a frame (resp. normalized tight fralne) for L2(d). A wavelet frame  is called 
semi-ohogonal if O,t 2 O.k for all k,f  Z d and m  n. It is called an 
onhonormal wavelet if {O.t} is an orthonormal basis for 
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\Ve define the dila.tiol Ulfitary operator D by 
(Df)(.r) = 
for f E L2(ld). Silnilarly, for any (s, t) E I d×d, the translation and modulation 
mfit.ary operators are defincd by 

alld 

for ail f  L(d). Thon D, E and T arc mfitary operators on 
Write lAA = {Ee, Te2 : (ÇI,(P2) ( A} alld /D.T = {D'"T« : rn e Z.t  e zd}. We 
will cab lAA (resp. IAV,T) a Gabor unitary system (resp. wavelet unitar9 system). 
It is easy to check by thc definition that the group generated by L/A is contained in 
"lFlAa, where "IF dcnotes thc llllit circle. 
hl quantmn and reprcscntation theory, a projective unitary representation 7r for 
a countable discrcte (hot necessarily abelian) group Ç is a nmpping g -- U from Ç 
into the set of mfitary ot)erators on a Hilbert space H such that UUh = l(g, h)Uh 
for ail 9, h E ç, where p(g, h) bclongs to the cir«le group "IF and (g. h) --, #(g, h) is 
called a multiplier of Ç ([Va]). It is easy to vcrify that a Gabor unitary system is 
the image of a project.iw.  Ulfitary representation t  -- U for the group A. In general 
for a countable set of unitary operatol's lA acting on a separable Hilbert space H 
that contains the identity operator, we will call lA group-like if 
group(lA) C "IFIA :-= {tU : t e "IF, U e lA} 
and if different U and I" in lA are always linearly indcpendent, where group(lA) 
denotes the group generated by lA. \Ve claire that a group-like mlitarv systeln 
is alwavs an image of a projective unitarv representation n for the group Ç := 
grovp(lA), hl fact, for an,v element V  Ç, by the definition of group-like unitary 
systems there is a unique element U  L/such that I" = tU for sonle t E "]I'. Define 
w(V) = U. Then I will be a projective unitary representation of Ç such that 
n(Ç) = L/. Typical exalnples of group-like unitary systems include group Ulfitary 
systems, Gabor Ulfitarv systems and Gabor-type Ulfitarv svstems (cf. [Ha], [HL]). 
For Gabor franleS we will prove: 
Theorem 1.1. Let A be a full tank lattice in ]R d x ]R d, and let g be a Gabor frame 
generator associated with A. Then S-1/2g is the unique best NTF approximation 
for g. where S is the frame operator for g. 
Theorem 1.1 is in fact a corollary of the following main result of this paper: 

Theorem 1.2. Let lA be a group-like unitary system acting on a Hilbert space H, 
and let b be a complete frame vector for lA. Then S-W2b is the unique best NTF 
approximation for b, wher« S is the frame operator for b. 

The proofs of Theoreln 1.1 and Theoreln 1.2 will be given in section 2. We note 
that Theorem 1.2 is not valid when the group-like unitary system lA is replaced by 
wavelet unitary systems as the following exalnple shows: 

Example 1.1. Suppose that g is an orthonormal M-dilation wavelet. Let (/,  
--- g. 
Then S-1/2!b is hot the best NTF approximation for /,. 
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In this example, although ,.--l/2j is hot the best NTF approximation among 
ail the normalized tiht wa.velet frames, it can be checked that it is the best NTF 
approximation alllOllg ail the normalized tight ones that are similar to . There- 
fore this example suggests the following modified NTF al»proximation problelu for 
wavelet svst,IIIS: 
Question 2. Let $ be a waelet fla.me for lo.T. Does there exist a normalized 
tight wavelet frame O such that 
(1) Ilç- 1 = '"l'{llh - '11" h  7(«».).h }? 
While we are mlal»le to give a flfll auswcr to this question, we will prove: 
Theorem 1.3. If  is a semi-orthogonal wavelet frame, then there exists a wnique 
noalized tight wavelet frame ç such that (1) hold.. Moreover, ç = S-1/. 
Iu section 3 we will prove Theoreln 1.3 and discuss SOlUe related inoblems. 
'2. GABOR FRAMES 
Iu this section we prove Theorem 1.2, which also implies Theorem 1.1. Since 
the basic techniques used in this section and the next section involve von Neumann 
algebra theory, we first introduce some notation. 
A von Neumann algebra , is a *-subalgebra of B() such that I  , and 
M is closed in the weak operator (or strong operator) topology. Bv the double 
COlnnmtant theorem, a *-subalgehra , of B(H) is a von Neulnalm algebra if and 
only if  = .", where  is the commutant of . A von Neumanl algehra 
is said to be finite if every isometrv in the algebra is unitarv. Two projections 
P and Q in a von Neumann algcbra  are said to be eq-uivalent if there is an 
operator T   such that TT* = P and ç*T = Q. So  is finite if there is 
no proper subprojection of I in  that is equivalcnt to I.  refer to [KR] for 
more information about von Neumann algebra theory. For a subset X of H and a 
subset  of B(H), we use IX] and w*() to denote the closed subspace generated 
by X and the von Neumann algebra genera.ted bv . respcctively. In this section 
we always assume that 11 is a group-like unitary system acting on a Hilbert space 
Lemma 2.1. Let  be a complete wandering vector for l¢. Then a vector    is 
a complete wandermg vectorforl¢ if and onlg if there is a (unique) unitary operator 
A  11' such that  = A. 
Proof. Let Co(l¢) := {T  B()  TU = UT, U  11}. Then it is a trivial 
exercise to check that CO (U) = 11  since 11 is group-like. Thus Lemma 2.1 follows 
from Proposition 1.3 in [DL].  
Even when 11 is a wavelet system (in this case Co(l¢)  11'), the above lemma 
still gives a parameterization of ail the wavelet.s in terres of the unitary operators 
in Ço(l¢). This simple observation is essential in the operator-interpolation wavelet 
theory [DL] developed bv Larson and Dai. However, in general, we do hot have a 
complete wandering vector for an arbitrary group-like unitary system 11. To prove 
Theorem 1.2 (also Theorem 1.3), the von Neumann algebra w*(l¢) will play a more 
important role than its commutant.  will provide a new parameterization for 
normalized tight ff'ame vectors by the unitary operators in w*(//). The special case 
when 11 is a group bas been discussed in [HL]. 
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Lelnlna 2.2. Let Jv[ be a von Ne'umann algebra on a Hilbert space  and let 
P  Jv[' be a projection. S'uppose that U  ]PH i8 a unitary operator. Then 
there is a unitary opentor 
Proof Let S  oe be a self-adjoint opcrator B su«h that U = e iB. Then there 
(A + A*). Note that 
is an olmrator A 
A*[pH = B* = B. [t follows that C]pH = A. Let II = e iC. Then li" G  is 
unita.ry and lipH 
Let H I)e a group-like unitary systcm on H. By definition, there exists a function 
f : group(14)  OE and a mal)ping a : group(H)  H such that 1I" = f(li)a(ll) for 
all I1" G 9roup(H). To sec that f and a are well defined, let ii" = aU 1 = 2U2 with 
U. U2  H and , 2  OE. Thon rl = U2 alid  = A2 since H is an independent 
set. Hcnce ])olh f and a m'(" wcll dcfincd, lsing this we can dcfinc the left (resp. 
right) regular repres«,ntation as in thc group case. 
Let 2(H) lw the tlill)ert st)ace of square-summal)le scquences indexed by H. and 
let { u } ho the standard (,rthonormal basis wherc kv takes value 1 at U and zero 
everywherc else. Vbr etch fixcd ( G H, we ch'fine Lv  B(12(14)) by the formula 
Thon L is a unitary rcprcscntation of on 12(H) such that Lc.L. = 
and L 1 = f(U -)L«(v-,) for all U. I"  H. In the group ce, this is exactly the 
left regular rei)resentation for thc group. Thcrefore we also call L the lefl regular 
representation for the group-likc unitary system H. Thc right regular representation 
Ru of H is (lefined 
Let . denote the von Neumann algebra generated by {Lv : U  H}. As in 
the group case, the commutant ' is exactly the von Neumann algebra generated 
by the right regular rcprescntation and both , and ,' are finite vn Neumann 
algebras ([CH1]). There is a uatural conjugate linear isomorphism  from  onto 
' dcfined by 
(A)BI 
In particular. (A)kI = 4"i for all A  «. For an orthogonal projection P in 
'. we use Lp to denote the subrel)reseutation of L restricted to the range of P. 
The following lenmm is the main ingredient in proving Theorem 1.2. 
Lemma 2.3. Let  be a complete normalized tight frame vector for a group-like 
unitary system  and   . Then 
(i)  is a complete normalized tight frame vector for H if and only if there ex 
a undary operator A 
(ii)  is a complete frame vector for l if and only if there exists a invertible 
operator A  w*(H) such that Aq 
Proof. We will prove (i), and the proof of (ii) is similar. Let , be the von Neunmnn 
algebra generated by {Lv : U  H}. Define T from  to g(H) by 
U 
Claire. T is an isometry such that TU = LvT and T = PXI, where P ls the 
orthogonal projection from g2(H) onto the range space of T and P  '. 
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This clailn can be checked by using the definitions of group-like unitary syst.ems 
and norlnalized tigllt frame vectors. To save space we leave it to the illterested 
reader or the reader can refer to [GH1] for details. 
By the above clailn,  is unitarily equivalent to t.he unitary systeln {L]p  U 
}. Therefore, without loss of generality, it suces to prove this lemnm for the 
case  = {Lu]p  U  11} and ç= P, where Pis an orthogolml projection in 
First aSSllllle that 4 G u'* () is unitary. Thcll, by Lelnllla 2.2, there is a llnitary 
operator B  M such that A = PBP. So A = PBP. = PBPk, = PBk,. Now 
for 311y X  Rage(P) we have 
=  I(*,PL,),  
=  I(P«,r(e*),)I  
=  I{,(*)Lc*w)]  
U 
=  [{(B*)*a, LaxI}I  
U 
= I1(*)*11  = IIll . 
where in the fourth equality we use the Net that (B*)Lff = La(B*), and in the 
lt equality we use the Nct that (B*) is unitary. Therefore ,4ç is a complete 
normalized tight fralne vector for U. 
Now let   Range(P) be a colnplete normalized tight fralne vector for U. Ve 
nd to find a unitary operator A  'w* (Ù) such that A = . For this purpose we 
define a bounded operator B on t(U) by Bv = Lv (U  U ). Let f and « be 
the associated nmppings determilmd bv the group-like unitary system U. Then for 
y U. l"  U we have Lt;L- = f(UV)L«(r;v). So 
BLcv = Bf(UV)g«(vv) = 
= f(UV)L«(uv) = LvL- = LvB-. 
Hence BLa = LrrB for all U  U, which implies that B  M'. 
It is routine to check that BB* = P. Therefore B is a partial isometry in 
Let Q = BB*. Then P and Q are equivalent projections in M'. Since M is a finite 
von Neumalm algebra, it follows that pZ and QZ are also equivalent (cf. [K]). 
Therefore there is a partial isometry C  ' such that CC* = P and C*C = 
Let T = B+ C. Then T is a unitary operator in M', and so P-(T)P is a unitary 
operator in W*(). g also have 

_ _ Prc-I(T)P\I = Prc-I(T)xI 
= Prc(rc--(T))',(.l = PT',iI = P(B + C).à.l 
= P( -J,- Ci). 

Since P{ = { and Range(C) = (I-P)f2(/d), it follows that A = {, as expected. [] 
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The following is a simple COllSeqllellce o[" the |»roof of Lelllllla 2.3 which will he 
us('(l il [he next 

Corollary 2.4. Let q be a «omplete 7urrntalized tight #ame vector for a grottp-like 
mdtary systern l,t. Suppose that ç E 7-{ su«b that bl is a normalized tight #ame 
for the closure [/'] of it.s linear spwn. Then tbere is a partial isometry A 
s.uch that ç = .4q. 

Pro@ As in the proof of Lelnlna 2.3, we only need fo cousider the case that ù = 
{Lc,[,p: U ¢/'}, i Ï = P\,, and   /i'«u.q«(P), where P is an orthogonal projection 
in .Ad'. Again let /3 be a ioulMed operator on g2(/¢) defined bv B\u = Luç (U  
l¢). Then /3  .Ad' is a partial isoluetrv. So n-(B) is a partial isolnetry in .Ad. 
v«hich ilnplies tha.t A := Pzr-(I?)P is also a partial isolnetry in w*(û). Moreover, 
a similar calculaion as in the proof of Lelnllm 2.3 shows A/Ï = P{. So .4q = { since 
 E R«,.qc(P). [] 

Proofof Theot'rm 1.2. Let S i»« the ri'ame «»perator fol" lit],.  first check that 
.s" E lt'. Let f and « iw the asso«iat(,d maplfiigs with 14. Then for any I E H and 
«illy .r E  WC haro 
By the definition of group-like unitarv systems, it tan be checked that {«(l'*u)  
çhrefore $ E '. Henc«, bv le sandard spectral decomposition theorem for 
positive operaor, $-/, S -/ E '. H,nc {$-/U   E } = {Uç-/ " 
U E }, whiH implis hat q := S-/ is a comple normalized tight frmne 
vector for . 
ow le ç be any compleç normalized tight frame vctor for . v Lmma 2.3 
there is a unitary operator A E w*() such that Aq = . So 
I1- 11  = I111  + IIlff - z(.) 
= IIA,lff + I111  -  e(.«,,  
= II"lff + I111  - 2 (.«,, ) 
= I1-/%11  + II'lff - 2 (A-/%, . 
Sinc AS -/  $-/, it. follows from he Cauchy-çwartz inequality that 
I(A-/,,,)I  I(Aç/.,,S-/%) I 
= I1-/,11 = -/,.,). 
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Therefore, 

llç - II = > llS-'/=ll= + IIII = - (s-I/2, ) = llS-/=- II =. 
So S-112 is the best NTF approximation for . 
For the uniqueness, assume that  is another best NTF approxilnation for . 
Then, again by Lelnma 2.3, there exists a Ulfitary operator A  w*(U) such that 
ç = AS-112. From iiç-¢ll = = iIs-1/u@-@liu , we have Re(ç,@) = (s-1/2,). 
Note that in the Çauchy-Schwartz inequality, the equality holds if and Olfly if there 
is a constant a such that AS-114 = ,S-114. This implies that 

No 

 = AS-112) _= S-114AS-1/4) = ozS-112. 

Since lai = 1, it follows from the al»ove identity that c = 1 

Thus  = .ç-l/2ç. [] 

Proof of Thcore'm 1.1. Let 7-/= L2(N d) and/d = HA. Then H is a group-like unitary 
system. So Theoreln 1.1 follows frolu Theoreln 1.2 innnediately. [] 

Re'mark. For a full rank lattice A in ]2d, we write A = IIZ 2a where M is a nOll- 
singular 2d x 2d real lnatrix. For Gabor systelns, one of the fundamental questions 
is: Under what conditions on A do we have a complete fralne vector for L/a? The 
well-known density theoreln (cf. [Dau], [aie], [[St]) tells us that one necessary 
condition is that I det M] _< 1. In the separable case A = L; x/C, where L; and/C are 
full rank lattices iii ]R a, D. Hall and Y. \Vang proved iii [HW1] that this condition 
is also sufficient by solving a problem in the geolnetry of nulnbers, motivated bv 
issues in harlnonic analysis. It still remains open whether this condition is sufficient 
in general. For lnore details about the time-frequency density we refer to [BHW], 
[CDH l, [Dau l, [DLL], IFS l, [GH1 l, [GH2 l, [GHa l, [GHL l, [Rie l, [lShl, [t2St], etc. 
Applying Theorem 1.2 to different unitary group systems we can get some inter- 
esting exalnples. ['or instance, 

Example 2.1. Let OE be the unit circle with the norlnalized Lebesgue lneasure # 
(i.e.,/,('iF) = 1). Let E be a measurable subset of "iF and let 7-/= L2(E). Çonsider 
the unitary group L/= {/1I,.,  m C Z}, where 11I I is the multiplication operator 
by the symbol f  L(E). It is easy to prove: 
(i) g is a complete normalized tight fralne vector for L/ if and only if 19(01 = 
1, a.e.; 
(ii)  is a complete fralne vector for L/if and only if 0 < a <_ I¢(«)l <- b, a. e., 
where a, b are constants. 
If S is the frame operator associated with a complete ri'aine vector , then 
Sf = z_,rn,a,XT" If eimt,/,eimtb.r, ,_ = Id12f. Hence S = ]lll,bl= and so S -1/2 = ]lll/lV I. 
Therefore Theorem 1.2 implies that I1 is the unique solution for the following 
nfininfization problem: 

Moreover, 

inf{llf - Wll- lY(t)l = , . e.}. 

inf{llf - Vil  If(t)l = , . e.} = IIZ'/IWl- Vif 
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Example 2.2. Let G l)e a discrctc group and M t)e the von Nemnmm algebra 
generat(,d i»v its h,ft regular rel»res('ntation. De'fine a ,(A) = (A, ) on M, where 
ç) = ke with e the id(,ntity eh'ment in G. Then  is a faithflfl trace an M, and 
the Hilbert-S('hmidt n(,,m AH8 is defin(,d bv A[ 2  
 ns = (A A). Let A bean 
invertible operator in M and let A = I'[A] be the l)olar decomposition of A. Then 
Theorem 1.2 implies the following: 
Iii'- Allus : ni,,{llU -AIIH : U  U()}, 
where U(,) denotes the set of all the unitary opcrators 
Remrk. Ater this palwr was finalized, we were informed that independently and 
at the smne rime, .I;mssen and Strohmer also obtained Theorem 1.1 for the one- 
dimensional case  when A = cZ x/3Z in [JS] with a completely different approach. 
However, our results hot only apply to general Gabor frames (including Gabor 
frames flr subspa«cs) but more generally apply to any frames induced by any pr 
jective mfitary rel»resentations and. in lmrticular, to frames induced by translation 
groups. , thank Professor Strohmer for sending us the mmmsçript [JS] and al 
thank professor tl. Fichtinger and M. Frank for bringing the work of .lanssen and 
Strohmer t our attention. 

3. ,VAVELET F'RAMES 
In this section we will mainly prove Theor(,m 1.3. We start with a few remarks 
and questims al»out frmne operators and dual wavelet frames. In the theorv of 
wavelct analysis one of the basic problems is to find characterizations for wavelet 
frmnes in terres of simple and practical conditions. While there exist character- 
izations for some special classes of wavelet fl'ames such as orthonormal wavelets 
and normalized tight wavelet frames (cf. [HW2]), this problem mav be completely 
intractable at this rime. A more practical problem [Cal] asked by P. Casazza is: 
Question 3. Characterize the wavelet frame such that it bas a dual that is also a 
wavelet frame. 

Let S be the ff'ame operator associate<t with a wavelet ffame %b. Then bv defi- 
nition {S-D'Tt{; : n  . f.  d} is the canonical dual for b. Therefore this is a 
wavelet frame if and only if S-D"Te{; = D"Try-Ib hohls for any n 
i.e., S -1 is in the "local commutmt" ([DL]) of btD,r at 
C,(btD.r) := {A e B(L2(lla)) : AD"Ttb= D'TtAb, n e 
In the case that b is a Riesz wavelet (i.e., {D'T6#} is a Riesz basis), the dual 
is unique. Hence in this case b has a dual which is a wavelet ffmne if and only 
if S -  C(blm.T). This condition is far from satisfactory since practically it is 
very hard to check the condition S -1  @(NO,T). Concerning our approximation 
problem we are interested in the following: 
Question 4. Is S-/{; always a wavelet franle? If it is not, under what conditions 
do we have this property? 
Note that S-i/# is a normalized tight, wavelet frame if S -/2  
Again this is hard to check. Actually, since CT,(btm,r) is hot a von Neumann 
operator algebra (it is a weakly closed linear subspace which contains manv inter- 
esting von Neumann subalgebras [DL], [Lal), we do hot evn k,,ow whether S -  
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Cg,(/A/D.T) ilnplies S -1/2 ( C%b(/-{D.T). Theorem 1.3 tells us that S -1/2 ( Cb([D.T), 
and so S-1/9b is a lOrlna.lJzed tight wavelet frame when  is semi-orthogonal. 

Proof of Theorem 1.3. We first check that S-1/2 is a norlna|ized tight wavelet 
fralne. Since {S-1/-DTe¢ : n, E Z, g E Z d} is a nornmlized tight frame for L2(lI{d), 
it suflïces to check that S-11-D"Te¢ = D'TtS-112.$ for ail n  Z and g  Z d. 
Write ll'n= [DTt¢], the closed subspace generated by {D"T«¢ : (  zd}. Then 
II; _1_ ll'm fol" m # , and {DTeb : /  Z d} nmst be a fralne for IIÇ because of 
the Selni-orthogonality of ¢. 
Let S, be the fralne operator for {DTr¢ :/' E zd}. Then for each f  II/, we 
bave 

sf = 

since f _L II , for n -J= m. Thus S = (,,ez S and hence S -1/2 = (D,ez S /2. 
Moreover, 

kZ a 

= D"SoT«b. 

So 
D-'S,,D' f = Sol 
for ail f  Il 0, which ilnplies that D-'S2I2D  = S /2 on II'0. 
Since {Te]wo " f  Z d} is a group, it. follows that sl/=Teb = TgS-11=b. There- 
fore, 

S-/'2-D'Ttb = SI/2DnTt = DnSI/2Tt 
= D'TtS/9b = DnTtS-1/2 
holds for all n  Z and   Z d. Thus s-tl2 is a normalized tight wavelet fla.n,e. 
Now we show that ç-1/2 is a best NTF approximation for  among ail the 
nornmlized tight wavelet fralnes that are sinfilar to . Suppose that  is any 
norInalized tight wavelet frame that is silnilar to . Then there is a bounded 
invertible operator B on L2( d) such that BDTt = DTeJ for ail n G Z,   Z d. 
Thus 
S-1/BDTt = S-1/DT = DnTeS-1/. 
Therefore S-/=B induces a silnilarity between the two normalized tight wavelet 
fralnes  and S-1/=J, and so it must be a Ulfitary operator (cf. [HL], Proposition 
1.9 (ii)). Let V= S-/UB. Then 
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(D"Te¢, D'T»¢) 

= (D'*TS'-/ub, 
= (D"T,S/uç,, D'"T.b'/u¢) = 0 

if m # n sitice I1 2_ Il',,. This implies that O is also senfi-orthogonal, and 
hen«e {TcO} is a normalized tight frame for spa{TeO}. Let P be the orthogonal 
projection front L2( d) oltto [i 0- Then PTe = TtP holds for every 
is an invariant subspace for {Te  g  Z}. This implies that {TPO" g  Z d} will be 
a normalized tight frame for P{TO}(C iii)). X also know that {T«S-I/2o  
t'  Z d} is a nornmlized light frame for tih, i.e., ,,-I/2ç i a complete normalized 
tight frame vector for the translation group {Te  g  Z } restricted to II. Thus. 
by applying Corolla.ry 2.4 to lhe unitary group {T«]Ib : g  Z}, there exists a 
partial isonwtry A  w*(Te]uq»  g  Z d) such that Pç = 4S-/2. In particular. 
Sa/4,4 = .4.S' /4 since S, c(mmmtes with Te]u for each g  Z d. Thus 
 11,4s-'/ll iIS-/«ll  IIS-'/ll iis-,/ll 

Therefore, 

IIç- 11  = I111  + I111  - 2 Re(,) 
= IIV-lç-/ll  + I111  - R(,¢) 
= IIS-/%ll  + I111  - 2 (p, ) 
 IIS-/ll + I111  -  R(S-/, ) 
= IIS-/%- 11 . 
Hence S-1/2 is a bes NTF approximation for  among ail the normalized t.ight 
wavelet frames that are sinfilar to . 
Finally we prove the uniqueness. Assume that  is auother NTF approximation 
for  in the problem. In the Cauchy-Schwartz inequality, the equality holds if and 
Olfly if AS-1/4 = oS-1/ for SOlne COlnplex uumber  G C. The saine argument 
as in the proof of Theorem 1.2 shows that a = 1. Thus Pç = S-/2, and so 
Ile$11 = IIs-l/2oll = Il011. This implies that PO = 0. Thus $ = S-1/2ç, which 
implies the uuiqueuess.  

Corollary 3.1. If ¢ is a Riesz wavelet such that S -1/2 E Cp(lJD,T ), then S-I/2l, 
is the unique best NTF approximation for ¢ among those normalized tight wavelet 
frames that are similar to , where S is the frame operator for ¢. 

Proof. From t.he proof of Theorem 1.3, it suflîces to point out that {TtS-I/2 ,} and 
{Te¢} are norinalized tight fraines for the closed subspaces they generate, where 
¢ , ¢. Since ¢ ,  (also S-1/2 ,  by assumption), it follows that {DnTe¢} 
(resp. {DnTtS-1/2g'}) is also a F{iesz basis for L2(] d) since  is a F{iesz wavelet. 
It is known that if a normalized tight frame is also a Riesz basis, then it must 
be an orthonormal basis (cf. [HL], Proposit.ion 1.9 (v)). Therefore {D'êTre} (resp. 
{D'TeS-1/2¢}) is an orthonorinal basis for L2(la). Hence {TeS-i/2I,} and {Te(p} 
are normalized tight frames for the closed subspaces they generate. [] 
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Let Ll be a unitary systeln such that Ll = LllLl0, where Lll and Ll0 are two unitary 
grout)s such that Lll çLlo = {I}. Such a Ll will be called an abstract wavelet system. 
A complote franm vector b for Ll is called seni-orthogon.al if ULl0b _1_ VLl0b for 
different U, I" G Lll- The proof of Theorenl 1.3 is clearly valid fol" the following nlore 
general result: 

Theorem 3.2. Let b be a s«mi-orthogonal complete frarne vector for an abstract 
wavelet system Ll. and S be the associated frame operator. Then S-t/ is the 
unique complete normalized tight frame vector such that 
holds for every nornalized tight frome vector  that is similar to ,. 
Pmof of Example 1.1. Let S be the ffame operator for . Then a. silnple argulnent 
shows that S = I, where I is the identity operator on L2(u). Thus S -/ = 4I. 
Therefore, 
IIS-'/ , - ll = 3ll,ll = 3/4. 
Clloose , to be nornlalized tight wavclet ri'an,es sud, that 
The existence of such a scquence can be eailv chccked. For instance, in the case that 
d=l a.nd A=°_. wecanchoosed,,--   koe, withE= ([-2,-)U[,2)). 
Then ¢ is a normalized tight vavelet frame (cf. [HL]). Thus, 
Thereforc S-/2 is hot the best NTF approximation for . 
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MEAN CURVATURE FLOW, ORBITS, MOMENT MAPS 

TOMMASO PACINI 

ABSTRACT. Given a compact lielnannian manifold together with a group of 
isomctries, we discuss MCF of the orbits and some applications: e.g., finding 
minimal orbits. We then specialize to Lagrangian orbits in Kaehler manifolds. 
In particular, in the Kaehler-Einstein case we filld a relation between MCF 
and moment maps which, for example, proves that the minimal Lagrangian 
orbits are isolated. 

1. INTRODUCTION 

If a given sublnalfifohI E iii a Rienlaimian manifoid (]il. g) is not lninimal, "mean 
curvature flow" (MCF) provides a canonical way to deform it. 
Ideally, the flow should exist until either a singularity develops, preventing fur- 
ther flow, or the subinanifold becomes minimal, hl this sense, MCF should be a 
useful tool in the search for mininlal submanifolds. 
A third, but exceptional, possibility that might occur is exenlplified bv the 
"translating solitons" in ': subnlanifolds which, under MCF simply flow by 
translation and thus never converge to a limiting object. 
However, MÇF is a diflïicult topic with many open questions. In particular, there 
is no general theory which can explain what will happen to all E C_ (M,g) under 
MCF, or classify which singularities can arise. One is thus forced to studv each 
case individually, or at best to look for classes of submanifolds which, under MCF, 
behave in the saine way. hlost often this leads to restrictions on the dilnension (e.g., 
curves) or codimension (e.g., hypersurfaces) of Z, or on properties of the immersion 
(e.g., convex). 
In general, the presence of symmetries in a problem reduces the number of vari- 
ables, hopefully making things easier. Regarding MCF, the "'best" case is when Z 
is the orbit of a group of isometries of (BI. g). It is sinlple to show that, in this case, 
ail Nf obtained by MCF are also orbits, and MCF basically reduces to solving an 
ODE on the (finite-dinlensional) space of orbits. 
Group actions have been extensivelv studied. In particular, orbits of a (compact, 
connected) Lie group G acting on a (conlpact, connected) lnanifold I tan be 
classified into three categories: "principal", "exceptional" and "singular". This 
yields a simple and pretty picture of the geometry of the orbit space BI/G. 
The first goal of this paper is to fit MCF into this franlework, analyzing "what 
happens" to a principal (or exceptional, or singular) orbit under MCF. 
The final picture, presented in Theorem 2, constitutes, for several reasons, a 
good "example zero'" of MCF. It is simple; it generalizes the standard example of 
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thc "shrinkilg slhero'" in IRn; itis (co)dim¢,nsi, m-indel»end«nt; and especially, in 
the orifil setting, "cvcrything we might want to i»e truc for MCF, is truc". 
' thon restrict mr attention to Lagrangian orbits. Using IllOment nmps to "get 
a grasp on lht'lli", we pmve that tho set £(M: G) of points belonging to Lagrangian 
G-orbits constitutes a smooth suhnmnifold in any compact Kaehler ambient space. 
When (kl, g) is a Kaehler-Einstein (KE) manifold, it was already known that the 
Lagrangimi condit ion is prcserved under MÇF; we give an independent proof of this 
in thc orbit setting, and are then ff'ce to a.l>ply Theorel 2 to studv how Lagrangian 
whits ew»lve under ICF. As a simplo corollary, we find that "backwards MCF" always leads to a minimal I,agrangian orifit. 
Futaki l»roved lhal compact positive KE mauifolds corne with a "'canonical'" 
lllllllPlll mal» I*- In l'rq»ositùm 5 we show that l* is intimately related to the mean 
Clll-Wlllli-P of Lagrangian ¢rlfils. As a «orollary. we fiud that minimal Lagrangian 
Cwlfits (wrl fixed G) are isolaled. 
As alrt,ady nole¢l. ICF of Lagrangian sulmmnifilds is hot a new subject; there is 
also some ovorlap, in thc case of lorus aclicms, wit h [G]. l lowcver, given the number 
¢»f known KI manifi»lds wilh large ismlotry groups, there seems to bc no a priori 
reascm to limit oneself to tori. ()ur altcml»t is fo develop a "'complote" picture of 
the geu,ral G case, relying only on the l»asic tools provided bv the general lheory 
¢f G-actions, moment maps and trasformtttioll groups. In this sense, we are hot 
aware of anv seri«ms overlap with existe,ni literature. 

2. SMOOTtl (_;ROI.'P ACTIONS ON MANIFOLDS 
This section is mostly a review of standard fa«ts regarding nmnifolds with a 
group action.  refer to lAI for fm'ther details. 
 adopt lhc following conventions. 
 3I is a compact. COnlmcted, smooth manifold. Diff(M) will denote ils group 
of diffcomorphisms. 
 G is a compact, connected, Lie group acting on 3I; i.e., we are given a 
homomorphism i  G  Diff(M). The action of g  G on p  M will be 
denotcd 9 "P- 
The action is "effective" if i is injectivc. Notice lhat, since Ker(i) is normal in G, 
we mav "'reducc" a.ny G-action 1o an effective G/Ker(i)-action. 
Whenever a groul) H is hot connected, H ° will denote the connected component 
containing the idontity element. 
The action of G on 3I induce8 an acti¢m of Gon TM. If X  TpM. it is defined 
as follows: 
g. x := .](x)  
vhere 9. denotes the differenlial of the map g = i(g)  M  M. 
Letting  denote the Lie algebra of G, anv X   induces a "'fimdmnental vector 
field" ," on M. defined as follows: 
d [p(tX). p],=0 
2(v) :=  
where exp(tX) denotes the 1-dimensional subgroup of G sociated to X. 
For ail p  M, we define: 
G . p := {g . p " g  G} ç M "orbitofp(wrtG)", 
Gv := {9  G " 9 " P= P} S G "stabilizerofp(wrt G)". 
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Notice that Gp is a closed subgroup of G and that G.p ": G/Gp. A generic orbit 
will often be denoted O. Thanks to the compactness hypotheses, every orbit O is 
a smooth embedded sublnanifold of M. 
Notice also that Ghp = h  Gp  h-l; i.e., Ghp is conjugate to Gp. Thus. to each 
p  M we nmy associate a conjugacy class of subgroups of G: 
O  (Gp) := conjugacy class of the stabilizer of any p  O. 
This class is called the "type" of O. 
Let O be any orbit and p  O. Let  := TpM/TO. The action of G on TM 
rtricts to an action of Gp on TpAI, and TpO  TpM is a Gp-invariant subspace. 
Thus there is a natural action of Gp on V. 
This induces an action of Gp ou G x V, as follows: 
h.(g,v):=(gh-,h.v). 
Let G xG V : (G x V)/G denote the quotient space. Then G xG V is a vector 
bundle (with fiber V) over G/Gv  0 and there is an action of G on G xa V as 
follows: 
.Çl" [g2, V] :: [gl g2, V]. 
The following result shows that G x 6 V contains complote informatiou on the local 
geometry of thc group tion near O. 
Theorem 1. Let G, I be as above. 
Then there exist a G-invariant neighborhood U of 0 in M and a G-invariant 
neighborhood II of the zero section of G xG I such that U  G-eqivariantly 
diffeomorphic to Il'. 
Corollary 1. Let I. G be as above. 
(1) For each fixed ty. the union of all orbits of that type forms a (possibly 
disconnected) submanifold of M. 
(2) There is only a finite number o] orbit types. 
(3) There is an orbit type (P) wbose orbits occupy an open, dese, connect«d 
subset of I. 
The types of the G-action can be partially ordered by the following relation: 
If a given orbit  bas type (h'), any nearby orbit ' C G xh- V can be written 
' = G-[1, v]; it is simple to show that the stabilizer of [1, ri is the stabilizer K 
of v  V wrt the h'-action; so it is a subgroup of h'. In other words, type(')  
type(O). 
In particular, the type (P) defined by Crollary 1 must be an absolute minimum: 
(Æ)  (h), for  types (h-). 
It is also clear that dira '  dira  (the dimension of orbits is a lower-semicontim 
uous function on I) and that orbits of type (P) bave maximum dimension among 
ail orbits. 
The final picture is thus as follows. 
Given I, G as above, there are three categories of orbits: 
(1) "Principal orbits', corresponding to the minimal type (P). 
They occupy an open, dense, connected subset of I. 
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(2) "Exceptional ol'lits", corresponding to those types (K)  K/P is finite. 
Via the projection G X,c U  G/K, any nea.rby principal orbit, is a finite 
covering of the exceplional orbit G/K. 
In parti«ular, exceptional orbits and principal orbits have the saine di- 
mension. 
(3) "Singular orbits", COl'responding to those types (K)" diln K > dira P. 
Their dimension is stri«tly slnaller than that of principal orbits. 
Example 1. S 1 acts isometricallv on S 2 := {.r = + y= + z = = 1} ç N a by rotations 
along the z-axis. 
The orbits are the sels ri=  w«, where w« := {z  c}. The singular orbits, of 
type (S), are the poles; all other orbits are principal, of type (1). 
Exalnple 2. (In 5 "2 there is also an isometric Z2-action that identifies antipodal 
points. Since the two actions commute, the Si-action psses to the quotient 
There is one singular orbit, representcd bv N 2  l; one exceptional orbit, repre- 
2 
sentcd by $20 (the "equatof); ail other orbits, represented by 5 «, 0 < c < 1, 
are principle. 
ToEether, principal and exçeptional orhits constitute the set of "regular orbits'. 
Any reular orbit  = G  q, q  [« is the image of an immersion 
Notice that. 
exceptional, of type (I), then   GlU" and ç is a covering map of G/P over 
Y mav set [ := {p  [  Ç-p is a priucipal orbit} ç [ and, analogously, 
Each of these subsets, generical]y denoted /*, is a smooth submmfifo]d inside 
I and I«/OE also h a smooth structure. 
Thus the set M/Ç, which is compact and Hausdor with respect to the quotient 
topolo, h the structure of a "stratified smooth manifold', the smooth strata 
being the connected components of [«/G. Once again, ,[»/G occupies an open, 
dense, connected su6set of 
An interesting application of all of the above is the following, simple, fact. 
Corollary 2. Asue G a.cts on M, with prncipa tpe ( P). 
(1) If P 
Ths, if the action is eecive, P  {1}. 
(2) If pO 
Ths, 
Pro@ If P is normal, P  G, Vp  M . Since 1 is dense in [, it is ey to 
prove that P 
The proof of {2) 
Out lt goal, in this section, is to "understand" convergence of orbits. 
Assume Eiven a curve of prhlcipal or6its  (corresponding to immersions çt " 
Ç/P  
çç must distinguish three ces. 
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(1) 

() 

(3) 

Assume O - 4)  G/P  31 is principal, i.e., has minimal type (P). 
Since /I p" is open i11 /I, each orbit near O must also have type (P); 
thus, wrt the local linearization/I = G ×p I r based at O, P acts trivially 
on I" and G ×p V = G/P × V is the trivial bundle. In particular, this 
shows that Or --* 0 smoothly in/I, i.e., et --* ¢. 
Assume O - ¢ is exceptional. Then, near O, there are eit.her exceptional 
or principal orbits and they m'e coverings of O. It is still true that et --* ¢ 
smoothly, but the limit is hot injective. 
Assume (9 is singular. Let K be the stabilizer of p  (9. 
Locally, /I = G ×/,- I; Or = G. [l,vt] (for solne Vt Ç l ') and K,, _< 
K is the stabilizer of [1,vt]. Since (Kv) --- (P), all Kv bave constant 
dimension q. The corresponding Lie algebras t., are thus points in the 
Grassmannian Gr(q, :) of q-planes in t := Lie(K). By compactness of 
Gr(q, t), we mav conchlde the following: any sequence O C_ Or, O --* O, 
contains a subsequence O« such that t,« -- t0, for some t0 ff Gr(q. ). 
Let {X,... ,X} span a complement ofto in L and let {Iq ..... I;} span 
a complement of t in f. Then TpO is generated by the fimdamental vect.or 
fields ç), and T[1,v,,](gn is generated by -i and f). Since 2i are smooth 
on ]I and 2ilo = O, we sec that 112,o Il -' 0 (wrt any invariant metric 
on M). 
In other words, convergence to a singular orbit is described, up to subse- 
quences, by the vanishing of certain fundamental vector fields: which fields 
vanish depends on the particnlar subsequence. 

3. MCF OF ORB1TS 
Let us now fix a compact, connected, liemmmian manifold (M. g) and a com- 
pact, connected, Lie group of isometries, G <_ Isomg(M ). (P) will denote the 
minimal type of the G-action, and p the corresponding Lie algebra. 
Recall that, to anv immersion 0 " N  M, we may associate a vohnne 
vol(0) :=/r vole*g. 
Since any regular orbit (9 corresponds to an immersion ¢  G/P  M, we get a 
function 
ol'(/C  -- , O  vol(C). 
The quotient map re  M -- M/G yields a pull-back map re*vol " lI   N. 
We will often write vol((9) instead of vol(C) and vol instead of rc*z, ol. 
Proposition 1. The volume function bas the following properties: 
(1) vol  1lI   IR is smooth. 
(2) If bas a continuous eztension to zero on 11 sing. 
This defines a continuous function vol  llI ----, N. 
(3) The function vol 2  M  IR is smooth. 
Pro@ For any regular orbit (9 = ¢  G/P  Iii, ¢*9 defines a G-invariant metric 
on G/P. Let. Z ..... Z be any basis of T[I]G/P, induced by the projection onto 
g/p of elements Z Ç g  Z  p. Let Iz := Z' A-.- A Z be the induced left-invariant 
volume form on G/P. 
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Since both volume fin'ms are G-invariant, vol 9 = c.lt, for some c = c(O). Clearly, 
«=  .qij, where gij := ¢* 9[1](Zi,Zj). Thus 
"°'(°)=fa "°'« = /a  ' ' "=   ' L " 
/P /P /P 
Now let  be a curve of regular orbits, t =  : G/P  M. Assume that 
Or +O. IfOisalsoregular, O=ç:G/P M, thenpt :=¢t[1]+p:=ç[1]. 
Choose Z,   such that the induced fimdamental vector fields 2i span TvO. Then 
2i also span Tç, Or. Setting 9ij := o,1(2,2) and 9ij := 9](Zi,Zj), we find 
0. 
This shows that vol is smooth on kI . If O is singular, we saw in section 2 that, for 
any sequence O,, ç Ot : O,,  O, we lnW choose Zi so that. for some subsequence, 
  + 0; vol extends continuously to 
certain 2i vanish. This shows that 
so 
zero on I sin9. 
Since vol(O) 2 = det gj . constant, vol 2 is smooth on M. 
Corollary a (cf. [H]). Let G be an9 compacL conneced Lie 9roup actin9 b9 some- 
tries on a compact, conne«ted Riemannian manifold (M, 9)- 
Then there ezists a regular mivimal orbit of the G-action. 
Pro@ Since I is compact, the continuous flmction vol : kI   h a maximum, 
which necessarily corresponds to a minimal (immersed) orbit. 
Examp]e 2 of section 2 shows that the minimal orbit might be exceptional 
Let us now recall the notion of "mean curvature flow". 
Fix manifolds N and (M, 9), and an immersion : N  M. 
A smooth 1-parameter family of immersions t :   kl is called a "solution to 
the MCF of (N, ç)" if it satisfies the following equation: 
0 = 
where H(ç) denotes the "mean curvature vector field" of t, defined  the trace 
of the second findamental form of the immersion. It is well known that H is, up 
to sign, the "L2-gradient "' of the vohune fimctional on immersions: 

£ d 
,ot.(e,)l,=0 = - (/-t. 

Locally. (MCF) can be written as a II-order quasi-linear parabolic system of equa- 
tions. In particular, solutions always exist for some short rime interval t E [0, e) 
and are unique. 
We want to lotus on solving (MCF) under the assumpton that (E, ¢) is au orbit 
of a group of isometries. 
Consider the map H : p -* H, that associates to each p E M the mean curvature 
H, of the orbit G - p. This defines a vector field on M. 
The following lemma examines its contimfity/smoothness 
Lemma 1. Let H be deflned as above. 
(1) H la smooth alon 9 each submanifold 9iven b9 orbits of the saine t9pe. 
It is al,vo smooth on ]I . 
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(2) H is G-imariant. 

Pro@ The smoothness of H along orlfits of the saine type is clear. Smoothness 
on .àI rey cornes from the convergence properties of regular orbits: basicalb; H is a 
local object and dons hot notice the difference I»etween principal and exceptional 
orbits. 
(2) is a consequence of the fact that ail ingredients in the definition of H are 
G-invariant. [] 

In particular, H descends to a vcctor tield Oll M/G and is smooth along each 
stratum. If O = ¢ : G/K "--, ll and t' := ¢([1]), we can consider the following 
ODE on 
(MCF') { p(0)ib(t) == P-H[p(t)]' 
Notice that, given a sohltion p(t) of (M('F'), the G-equivariant map 
¢:a/lç×[0,«)-- at, ¢([gl,t):=g.p(t) 
solves (MCF) with the initial condition (E, ¢) = O. By uniqueness of solutions of 
(MCF), this shows that MCF of an orbit gives a curve of orbits. 
In other words, if (, ¢) is an orbit, (MCF) is equivalent to the ODE on 
(or on M) determined by integrating H. 
The reduction of the problem from a PDE to an ODE simplifies things enor- 
mously. For example, MCF of orbits bas the following properties: 
There exists a (mli(lue) sohltion Or defined on a maximal rime interval 
(a,/3): this cornes from standard ODE theory. 
(MCF) mav be inverted: i.e., t - Q(t) := O(-t) solves the equation for 
"backward MCF": 
d 
dUQt = -HQ" Qo = (9. 
This is truc for any ODE of the type à: = f(x(t)), but is very atypical for 
parabolic problelns. 
Another interesting feature of (MCF) on orbits is that it preserves types: 
Proposition 2. For each orbit (9. Ho is tangent to the submanifold determined 
by the type of (9. 
In particular. "if (Dt is the solution of (MCF) with initial condition (90 = (9. then 
type ((Dt) =- type ((9). 
Pro@ Let p  M and let G- p have stabilizer K. Locally near G- p, M = G x ¢ 1 
where V = Tp(G. p)± and K acts isometrically on V. 
This deternfines a decomposition of V into K-irreducible subspaces: 1" = 
Since H is G-invariant, it is also K-invariant: so H G 1 "° := {v G I" : k. v = v, Vk 
Notice that G x ,- V o corresponds to the orbits near G- p of type (K). Thus 
is tangent to the set of such orbits. 
Since this is truc for each p  M, (MCF) preserves types. [] 
Corollary 4 (cf. [HL]). Let (M, 9) be as abov«. 
If an orbit is isolated wrt ail other orbits of the saine type, then if is minimal. 
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We now Wmlt t.o show that, on M e9, (MCF) is a«tually a gradient flow; i.e., for 
some f  C(Meç), H = 
Let Pt I)c a c'Hrv(" in 1I re9 and Or := G'pt. We will let X d,note both the vector 
d 9 
Ptlt=O at Po and the G-invariant vector field aC tlt=o ahmg O. 
Sin«e H, X and the metric on 1I are G-invariant, (H, X) also is. Thus: 
dl. d 
This proves that 
d 
(H, S)po = - v°i(Pt)lt=° d 
'ol(po) = -lo9vol(pt)lt=o. 
In other words, H = -çio9(vol) on M''eo. 
 now have ail the information we need to understand how MÇF fits into the 
framework set up in section 2. 
Let O =   G/P  AI lc a fixed principal orbit and let Or = bt  G/P  M 
be the maxinlal curve obtained by MCF, with initial condition O(0) = O and 
Pt = ftt]. 
Since M is compact, there is a sequcnce {p} ç {pt} and   M such that 
pç. ThusO:=G-pO:=G-. 
In general, however, different sequences may have different limits; so we cannot 
hope that Or  . The following example of this w suggested to the author by 
T. Ihnanen. 
Exalnple 3. Consider an embedding s    {(a',B)   " z  + y > 1} that 
tends towards S t = {z  +  = 1} ç   t  , spiralling around it. 
Let S denote its image and f " S   be a positive, decreiug function on S 
suchthatf(t)c>0toe. 
S may be "fattened" by a tubular neighborhood if (of decreing width,  
t  oe). At ea«h point s  S, f may be extended, with constant value f(s), in 
the normal directions. This gives an extension of f  U   such that fls is 
tangent to S. A partition of unity argument now allows us to extend f to a smooth 
fHnctiOll ï" 2  . ClParly, s = f[s ad sx  c, 
Since we are only int.erested in what happens near S  , we may perturb  so that 
it extends to some compact (E , 9) containing a neighborhood of S . We have thus 
built a smooth gradient vector field on a compact nmlfifold whose flow does hot 
converge to a unique point. 
Now let M := E x S 1, with the obvious S-actiou. For each p  E, let {p} x S  
have an sl-invariant metric hp su«h that lo9vol({p}  S ) = f(p). 
For each (p, q)  M, let Tp,qM have the product metric 9] x hp[q]. 
Since Ho = -çlo9 ool(O) = -çf, MCF ofany orbit {s} xS  with s = S(to)  S 
yields the curve of orbits {s(t)} x S a, t  If0, oe). By construction, these orbits 
have no limit  t  . 
Ve are thus interested in conditions ensuring the existence of linto Or. 
The following lemma shows that, in the analytic «ontext, thiugs work uicelv. 
Lemma 2. Consider the ODE 
b = -çf, p(o) = po 
with matinal solution p(t), t  (a, ). 



MEAN CURVATURE FLOW, ORBITS, MOMENT MAPS 3351 

Assume that, for some subsequence t, -- /3, p(t,) -- y and that f is analytic in 
a neighborhood of y. Then p(t)  y. 
Pro@ Let s(t) := f: IIVf(pT)lld-r. Since [IV]'(pT)[I > 0, s is a diffeomorphism 
between (a,/3) and some (a', 
Since f(Pt) is monotone, it also gives a diffeomorphism between (a,/3) and some 
(a, b), where b = f(y). In particular, s can be written as a function of f E (a, b); 
i.e., s = s(f). 
Since  = V f- dp _ 
d-ï -- --IVfl 2, we find that dl _ 1 
ris IVfl and ris 
df -- Ivfl" 
It is simple to prove that p(t)  y <:/3' < 
We nlay assunle that f(y) = O. When f is analytic near y, the "Lojasiewicz 
inequality" asserts that there exists a neighborhood U of  and  > 1, c > 0 such 
that, on U, Ifl < e Içfl - Thus  , 
_  _< c. I fl-I and so 

If (M,g) is analytic and vol(Or) >_ c > 0, we may apply this lenmm to f := 
logvol, proving that (_9 + := limtz O(t) exists and is regular. We may then apply 
the following, classical lemma to N := M g. 
Lemrna 3. Let H be a smooth vector field on a manifold N. 
Let p(t)  t  (a,/3) - N be a maximal integral curve and assume that there 
exists q  N such that p( t )  q as t  /3. 
Then /3 = oe and H(q) = O. 
1 Vvot 2 In the analytic context, this 
Notice also that H = -½ Vlog(vol 2) - 2 ,.ol2 " 
shows that if Or  0 and O is singular, then IIHo, Il  oe (in the smooth category, 
following the idea of example 3, one could build examples for which such a limit 
does hot exist). The following examples show that (_9 may be minimal or not. 
Example 4. Let S  act on S 2 as in exanlple 1 of section 2. Any (_9 that is not 
the equator or a pole flows, under MCF, to the closest pole, which is a singular, 
nfinilnal, orbit. This happens in finite tinle. 
Example 5. The above action of S  on S 2 induces an action of S  on S 2 x S 2. 
Let (_9 _ S  x S  be the product of a "snlall" orbit in S 2 (i.e., near a pole) and 
a "large" orbit in S 2. The flow (Dt becomes singular as soon as the smaller orbit 
collapses onto the pole, but this limiting curve, p x S , is hot minimal: its flow 
exists until the second curve collapses. 
Summarizing, we have proved the following result. 
Theorem 2. Let (I, g) be a compact, Riemannian manifold and let G be a compact 
group acting by isometries on ]il. Let (9 be a priacipal orbit. Then: 
(1) MCF preserves orbits and types. 
Thus there exists a unique, maximal, curve of principal orbits (gt, t  
(,/3), solution of MCF with Oo = O. 
(2) Assume (M,g) is analytic and 0 + := limtz (Dt exists. Then: 
 0 + is a regular orbit <:/3 = oc <: [[H(t)[[ -- 0. 
In this case, (9 + is minimal. 
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 0 + is a smgular orbd ,  < oo  IIH(t)ll  c. 
In this case. 0 + mag bc nmimal or net. 
If t'ol(Ot)  c > O. thon 0 + ahvags eists aztd is 'gulav. 
(3) Assume (M.) is amdgtic. Then O- := limt Or alwas ezists. It is a 
ninital regttlar orbit.  = -oe ad ]]H(t)]]  O. In partictlar. "'backwards 
MC" alu,ays leads fo a minimal regular orbit. 
To get an almlogous statement fir flovs of exc«ptional or singular orbits, it is 
sufficieut to apply tlw theol'elU to the (Sluooth, compact) malfifold M' defined  
the closm'e in ,11 of the set. of orbits of the type in question: these orbits will be 
the principle orbits of the induced G-action on M'. 
Remark. Using "equivaria.nt hh,rse theory" applied to the wlume function, it would 
be interesting to studv the topology of Rielnalmial G-lnanifolds in terres of its 
lnininml Ol-bits. In theory, Theol«ln 2 would be usefill in this. 

4. LAGRANGIAN ORBITS AND MOMENT MAPS 
We uow want fo focus on Lagrangian orbits gencrated bv isometry groups of 
compa.ct Kachler manifolds. We start by reca.lling a few well-known facts concerning 
t.ralsfornlation groups of Rielmmllian ami Kaehler manifolds. \Ve refer to [K1] for 
proofs and further details. 
Definition 1. Let (M, 9) be a Rielummial lnanifold. A vect.or field X on M is an 
"iufinitesimal isometry" if £ \'9 = 0; equivalcntly, if the local flow generated b3 X 
is a curve of isolnetries. 
i(M) will denote the space of ail infinitesimal isometries. When (M, 9) is com- 
plote, i(M) is the Lie algebra of Isom9(M). 
Definition 2. Let (M, J) be a complex manifold. A (real) vector field X on M is an 
"infinitesimal automorphism" if £xJ =- 0; equivalently, if the local flow generated 
by X is a curve of automophislnS of (M, J), or if X - iJX is a holomorphic section 
of T'° M. 
tf(M) will denote the set of infinitesinlal automorphisns. It is closed «rt J and, 
v«hen (M, J) is compact, it is the complex Lie algebra of the group Autj(M) of 
automorphisms of (M, J). 
Theorem 3. Let (M. J, 9, ) be a compact Kaehler nmnifold. Then an!l infinitesi- 
mal isometrl is an infinitesimal automorphism: so Isom9(M) ° < Autj(M) °. 
This, in turrt, implies that Isom9(M) ° < Aut,(M) °. 
The following proposition, although very silnple, is the key to understanding 
Lagrangian orbits. 
Proposition 3. Let, (iii 2n, J,g) be a compact Kaehler manifold and let G <_ 
Isom9(M) act on M with principal t!lpe (P). 
Assume there exzsts a re9ular La9ran9ian G-orbit. 
Then P is finite; so dira G = n and p = Lie(Gp) = {0}. Vp G 11I «9. 
Proof. Assume O is a Lagrangian orbit. Then J gives an isomorphism TO ± - 
TO  I/P that is equivariant wrt the natural P-action. Notice that, for each 
p G P, this action coincides with the differential of the map 
p" /P -- a/P, p[] := [p] = [pp-'l. 



MEAN CURVATURE FLOW, ORBITS, MOMENT MAPS 3353 

In other words, the action of p on 9/P is the differential of the adjoint action of p on 
G/P. Takcn ail together, these maps form a group homomorphism P  GL(9/p); 
thc corresponding Lie algebra homomorphism is the map 
If (9 is principal, the P-action on TO ± is trivial. Thus P acts trivially on 9/P (i.e., 
the action of each p E P on 9/P is the identity). So the map p  91(9/P) is trivial 
(i.e., the action of each X E p is the zero map), i.e., p is an ideal of 9- This implies 
that pO is normal in G and Corollary 2 of section 2 proves that P is finite. 
Now assume (9 is an exceptional Lagrangian orbit of type (K). Locally, Iii = 
G ×/« V and K acts as a finite group on V =- 9/[!; so a neighborhood of 1 G/( acts 
trivially on 9/[!. This shows that/(0 acts trivially on 9/[!. 
As above, /(0 is normal in G. Since K ° = pO, pO is also nornlal and we may 
conclude as above. [] 

It is now convenient to introduce the concept of Hanfiltonian group actions. 
Again, we refer to lAI for flrther details. 
Let (iii, w) be a synlplectic nlanifold. Recall that a vector field X on Iii is 
"'Hanfiltonian" if w(X,.) is an exact 1-fOrlll Oll Iii; i.e., w(X,.) = d f, for some 
f  COe(M). We say that f is a "Hainiltonian flmction" for X. 
Definition 3. The action of G on 5I is "Hamiltonian" if the following conditions 
are satisfied: 
(1) There exists # : Iii --- 9" such that {d#[p](-),X) = w[p](«Y,-), where 
(-,-) denotes the natural pairing 9" × 9 --- l. Equivalcntly, VX G 9, )ç is 
Hamiltonian (with Hamiltonian function p\- := p H {#(p), X)). 
(2) # is G-equivariant wrt the G-action on Iii and the co-adjoint G-action on 
9*- Equivalently, #[X.Yl(P) = [pl(2, 
We say that p is a "moment, map" for the action. 

Remarks: 
(1) Assume (iii, J,g,w) is a compact Kaehler manifold and that, for some G <_ 
Isom(M) °, condition (1) above is satisfied. Then, VX  9, 
dx = (R,.) = g(JR,-). 
This shows that Vpx = J-Y, and so V#x is an infinitesimal automorphism 
of (iii, J). 
(2) 13y definition, the differential dp[p]  TpM -- 9" is the dual of the map 
dp[p]*  9 --- (Tpiil)*, X --- d#x[p]. 

Thus Imd#[p] = (Kerd#[p]*) # : (gp) #, where 9p = Lie(Gp). 
In particular, d #[pi is surjective iff Gp is discrete. 

Definition 4. 2 ç (iii, w) is "isotropic" if w] -- 0; if dira E = , and dira iii = 2n, 
then isotropic submanifolds are called "Lagrangian". 
We are mainly interested in moment maps for the following reason. 
Lemma 4. Assume the action of G on (iii, w) is Hamiltonian, with moment map 
#. Let p  iii. Then the followin9 conditions are equivalent: 
(1) # is constant on the orbit (9 = G . p. 
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(2) (,_9 is isotropic. 
(3) ,.(p) e [0,0] #. 
We now have ail the elemenls necessm'y to prove the following: 
Corollary 5. Let (AI, J, g) be a compact Kaehler manifold. Assume that G < 
Isomg(M) acts in a Hamiltonian fashion. Then the set 
/(AI: G) := {p G M'eg " G .p is a Lagrangian orbit} 
either is ernpty or is a smooth subrnanifold of AI reg. of dimension 2n - dirai0. 
Proof. If lhele exists a regular Lagrangian orbit, lhen, by Proposition 3, P is 
filfitc and dim G = . Thus every regular isotropic orbit has dimension n and 
is Lagrangian. Lclmna 4 now shows thaI, if wc let Preg denote the restriction of 
p t.o AU e, then £(AI;G) = p,-e([0.0]#). Since Op = 0. tt,« is a submersion; so 
£(AI, G) is Slnooth, of dilncnsiol  + dira [0- 0] #- [] 
Example 6. Assmnc G <_ l.omg(AI) is selnisimple. Then the G-action on AI is 
Halniltonial (cf. lAI) and Lagrangian orbits are isolated. 
Example 7. Assulne that a torus T' <_ I.omg(AI) acts cffectivel.v on AI and that 
HI(M;IR) = 0. Then thc action is Ha,niltonian (cf. lAI), P = 1, [0-0] = 0 and 
cvery rcgular orbit is Lagrangian. In other words, I:(M, G) = AI g. 
In parti«ular, there exists a minimal, Lagrangian orbit (cf. also [G]). 
An example of this is provid«,d bv the standard T'-action on I '. 

5. MCF OF LAGRANGIAN ORBITS IN KE MANIFOLDS 
In this section, we will aSSllllle that (M, J, g, w) is a KE manifold. 
Since we are interested in group actions, we lnust recall (cf. [K1]) some basic 
facts concerning their transfornmtion groups. 
Theorem 4. Let AI be a compact KE manifold such that Ric = c. 9, c > O. 
For any (real) vector field X. let Z := X - idX and ( := 9(Z, "). Then 
(1) i(M) is totally real in h(AI): i.e., if X  i(M), then dX  i(M). 
(2) X e b(M) +, Ç= Of" f  C(M;C),Af = 2cf. 
In particular, f^t f = O: so such anf is unique. 
(3) X  i(A/) ¢, Re(f) = 0. 
If we set E2 := {f  C(AI;I)  Af = 2cf}, there is an isomorphism: 
E2oei(M), f  i-f = Ç. 
(4) D(M) = i(Al) @ Ji(Al). 
It is possible (cf. [I£2]) to prove that positive compact KE manifolds are sim- 
ply COlmected. This ilnplies that every fundanental vector field induced by G < 
Isom(A1) is Hmniltonian. From our point of view, however, much more is true: 
Proposition 4 (cf. IF]). Let M be a compact positive KE ,aniyotd and G <_ 
Isomo(A1). Then the action of G on M is Hamiltonian. 
Recall the correspondence and the notation from Theorem 4 above: 
.\'i(Al),-f'fCoe(Al;l), Af=2cf, Ç=iOf. 
Then #x :--- -½ f defines a moment map for Al, G. 
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Moment maps are usually hot unique: if p is a moment map, p + c also is, for 
any c E [9, 9] # <- 9*- The proposition above suggests the following. 
Definition 5. The moment map defined in Proposition 4 above will be called the 
"canonical moment map" of the G-action. 
Recall, however, that moment maps are uniquely defined on [9, 
= w(X, Y). Recall also Lemma 4. Proposition 4 thus leads to the following result: 
Corollary 6. Let M be a compact KE manfold sch ttmt Ric = c. 9, c > O. 
() vx,  i(t), (x,r)  . 
(2) çf G E2c, f stricted to £(M; G) is G-invariant. 
Putting everything together and using the fact that KE metrics are a.nalytic, we 
can now prove the following result: 
Theorem 5. Let M be a compact positive KE manifold and let G  Isom(M). 
Assume £(M:G) is hot empty. Then H i tamgent to £(M;G). Thus MCF 
pserves the Lagrangiam condition and may be studied  in Theorem 2. 
Furthermore, there exists a minimal Lagrangian orbit. 
Pro@ Recall, for anv Lagrangian submanifold E immersed in Kaehler M, the 
isomorphism 
(T)  (), :=(ç.),r. 
It is well known (cf. [TY]) that if aH  A(E) denotes the 1-form corresponding to 
the mean curvature vector field H under this isomorphism, then d aH = pr, where 
p(X, ') := Ric(JX, Y) is the "Ricci 2-form'" of 
When I is KE, p = c. w; so this shows that aH is closed. 
Now let p e £(M;G). Then Tp£ = {X ¢ TpI" d,,l(x) e [,l#}. So we 
hem to prove that «,l() e [. 1#. 
Since H is G-invariant, CH also is; i.e., CH  9*- Notice that dp](H) = 
](-, H) = 
Recall that, for any 1-form   A  (E), 
Thus 0 = daH(X. 1") : -aN[X, Y], VX, 1  9, as desired. 
The first clailn is now obvious. The properties of vol show that there is a 
Lagrangian orbit O of maximum vohune (which is minimal in (1; G)). Let Or be 
- fM (Ho, Ho); so Ho  O. 
Remark. When I is compact Kaehler Ricci-flat., one can show that Isom(I) ° is 
a torus. So example 7 shows that the analogous statement is triviallv truc. 
When AI is compact negative KE, Isom(BI) ° = {Id}. So these mani%lds are 
hot interesting from our point of view. Cf. [K1] for details. 
Out final goal is to explore the relationship between MCF and the canonical 
moment map. 
Proposition 5. Let I  be a compact KE manifold such that Ric = c. g, c > O. 
Given G  Isom(M), assume that {M;G) is hot empty. Let 
denote the canomcal moment map. Then, on (BI; G), 
(1) VX9, Ho'Vpx=cx. 
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(2) Vp E £(M;G), the naturai (G-invariant) metric on G.p ç bi &fines 
metrcs on  and *. For the induced norm (which depends on p), 
dll,ll=](H) = 2cll.(P)ll =. 
Proof. Let 0 
TvO be an orthonorllml basis wrt the induced metric. To simplify the notation, we 
will dcnotc the corrcsponding fimdamcntal vector fields also by e. Then 
(Ho, 
section 4, we saw that ItX is ail infinitesimal automorphism of M Thus 
ÇJe Çpx = 
The definition of thc canonical moment mal) now shows that 
2(Ho, çp x) = -(e, ççpx) - (,lej,,l(VVpx)) 
= -divM(çItx-) = MPX = 2Cpx. 
This proves (1). AI)plying (1) to X = ei, nmltiplying by 2p.e and summing wrt i 
shows that 
which is (2). 

We can now prove 
Theorem 6. Let M 2n be a compact KE manifold such that Ric = c. 9, c > O. For 
G <_ Isom9(lI ), let p. d«not« tbe canonical moment map and let E2c(G) := {f E 
E2c " f = p..\', for some X  9}. 
Assume that regular orbits bave dimension n. Then a Lagrangian orbit (.9 is 
minimal iff it(O) = O. In particular, minimal Lagrangian orbits are isolated. Fur- 
thermore, the followin9 are equivalent: 
(1) There exists a Lagrangian orbit. 
(2) There exists a minimal Lagrangian orbit. 
(3) 0  #(M). 
(4) The set {p  M" f(p) = O, Vf  E2«(G)} is not empty. 
Proof. By hypothesis, an orbit is regular iff it is n-dimensiolml. In particular, every 
Lagrangian orbit (_9 is regular. We may thus restrict our attention to M'«9. 
If (_9 is minilnal, Proposition 5 shows that p(O) =- O. Vice versa, assume that 
p.(O) = 0. Let Or be obtained by MCF applied to (_9. Then Proposition 5 shows 
that f(t):= II.ll=(o) satisfies 
f(t) = 2cf, f(O) 
This implies that f(t) =- 0; so (_9(t) C #-(0). However, p. is a submersion; so 
p.-(0) is smooth of dimension n and, since P is finite, the elements of p.-(O)/G 
are isolated. Thus (_9(t) = (_9, i.e., (_9 is minimal. 
Together with Theorem 5, this shows that (1), (2) and (3) are equivalent. The 
equivalence of (3) and (4) cornes directly from the definition of p.. [] 
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Remark. In the toric case, one can show that if-l(0) is connected. So Theorem 6 
implies that the minimal Lagra.ngim orbit, is unique. This result was obtained also 
in [G], by lifting the T'-action from ]1I to its canonical bundle Iç^i and studying 
the induced geometry. 
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SINGULAR INTEGRALS ON SYMMETRIC SPACES, II 

ALEXANDRU D. IONESCU 

ABSTRACT. We extend some of out earlier results on boundedness of singular 
int.egrals on symmetric spaces of real rank one to arbitrary noncompact sym- 
metric spaces. Our main theorem is a transference principle for operators 
fined by IK-bi-invariant kernels with certain large scale cancellation properties. 
As an application we prove L p boundedness of operators defined by Fourier 
multipliers that satisfy singular differential inequalities of the H6rmander- 
Michlin type. 

1. INTRODUCTION 
Let G be a noncompact connected semisimple Lie group with finite center, K a 
maximal compact subgroup and X = G/K an associated symmetric space. In this 
paper we stu(ly L p boundedness properties of a class of operators on the symmetric 
space  which are the analogue of the singular integral operators on Euclidean 
spaces. By analogy with the Euclidean case these operators can be defined by 
Fourier multipliers or by convolution with CalderomZygmund kernels. 
A necessary condition for L p bonndedness of an operator T defined by a mul- 
tiplier m is that the multiplier extends to a bounded IV-invariant holomorphic 
function in the interior of the tube  = a* + ico(W- pp), where pp = 12/p - llp. 
This was observed by Clerc and Stein [5], who also proved a sucient condition for 
L p boundedness when the group G is complex. Bv analo with the Euclidean case, 
a natural theorem is the following: sume that p  (1, 2) U (2, oe) and the multi- 
plier m extends to a holomorphic function in the interior of the tube . Assume 
in Mdition that m satisfies differential inequalities of the form 
for ail A   and any j  0, where m(A) denotes any partial derivative of 
m of order j. Then the operator T extends to a bounded operator on LP(). 
Statements of this type in various settings can be found in [5], [18], [4], [22], [1], [8] 
d [91. 
In this paper we deal with operators defined by multipliers that are allowed 
to have singularities on the boundary of the tube . These multipliers satisfy 
differential inequalities of the form 
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wlwre the flmclion d(A) can wmish on th«, I,,,mdary ,,f the tul,e  (as in (4.1)). 
These difli'rential in,qualities are the analog m symmetric spaces of the clsi- 
cal tliirmander Mi«hlin difl'rential inequalities. Natural examples are provided 
Iw imaginary powers of a suitalfly modified Laplacian, « in [8]. The associated 
multipliers have singularities a.l the points {w-(ipp) : u'  II'}. 
The main diffi«ultv in proving L p bomMcdness of operators defined by multipliers 
with singularities on Ihe bomMary is /hat the classical tterz majorizing principle 
[13] fails fo alply. As in thc çase of E('lidean spaç«s, il is critiçal to be able t.o use 
the large scale can«ellatim prq«rlies of tlw kern«! of such an operator. Onr main 
lheowm in Ihis dircctim is Th«'»rem 3. l -a Iransfercnce principle that allows one 
fo deal with çonvolutim op«rators dcfined ly kernels of the fonn 
(1.2) K (exp tl) = e -°(H)/P(H), 
where  is a ('aldmon Zygmml kernel on o. llere I < p < 2 and the Nctor 
«,-o(u)/p is Ihe critical CXl«m«ntial dccay factor for L p boundedncss. The proof 
of this Ihcorem is simila.r to Ilw lroof of Ihc corresponding theorem on symmetric 
spa.ces of real tank o,,e ([1-1. Th,,,,,'en, 11). 
()m" s(,«(md mai tsk is Io show that this transfcwnce principle, together with 
the ]Ierz majorizin principl(, and local analysis, succs to In'ove L p boundedness of 
operators defined by multiplicrs satisfying (4.1). This is significantly barder in the 
case of gencral svmmctriç spaces thal il thç case of real tank ont symmetric spaces. 
()ne of Ih(, di«ulties arises from thc fi«t that thc Harish-Chandra expansion of 
the elem(.ntarv pherical flmctions does hot converge mfiformly in the region close 
«» the walls of the positive kkbyl chamber. Thus one cmmot get good pointwise 
estimates on the kernels of the operators in this region. This difficultv w overcome 
bv Anker [1] and we follow his approach. The main observation we need to make is 
that tbe lmne of this region is smaller than the vohnne of the flfll kbyl chamber 
(in tire sense of (4.6)). k will thçn apply out transference theorem to the part of 
t he kern«l supported away ff'oto the walls of the positive kvl chmnber. The Harish- 
('hmMra xI)ansion converges unifonnly in this region and the kernel corresponding 
to the main tenn in this expansion is of the fonn (1.2). Also, one can use the Herz 
majorizing principlc to deal with the error tenns in the Harish-Chmdra expansio 
in this region. 

"2. PttELIMINARIE 

Most of out notation rela.ted to scmisiniple Lie groups and svmmetric spaces is 
standard and can be round, for example, in [12]. Let G be a noncompact colmected 
selnisimple Lie group with finite conter, 1 the Lie algebra of G, 0 a Caltan invohltion 
of 1 and 1 --- tÇ p the associated Crtan decomposition. Let ]K = exp t be a maximal 
compact subgroup of G and let  = G/]K be an associated symmetric space. Let a 
be a maximal abelian subspace of p, NI the centralizer of exp a in ]K, E the restricted 
foot system of the pair (1. a) and lI the associat.ed Weyl group. Let a + C a be 
a positive Wevl chmnber and + the corresponding set of positive roots. For any 
root. c  }2 let , be the foot space associated to a and mc its dimension. Let 
n = -aez+ ça, g = 0(n), N = expn and N = expg. 
The group G has an Iwasawa decomposition G = K(exp a)N and a Cartan de- 
composition G = 1K(exp a+)lK. For each g  G denote by H(g)  a and g+  a + 
the middle components of g in these decompositions. We will also use the Iwasawa 
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decomposition G = N(exp a)K. We normalize the Haar measure dk on K with total 
mass 1 and the Haai" measure d-g on N such that 
e-2p(H(5))d = 1, 

where 

1 
p(H) =   .(H). 

cE+ 
The Haar measure dg on G tan be normalized so that 

(2.3) 
where 

£ F(g)dg -- N  /E F(g(exp H)k)e2p(H) dkdHd- 

and 
for any continuous coinpactly supported fuuction F : G -- C Olm has by definition 
(2.1) (H) = C H (sinha(H))'«" 
The Killing form of 9 induces a scalar product  , ) on c. DeltOte by a* and « 
the real and the COlnplex dual of a. For ally  G a* let HA be the torique element 
of a with the property t.hat HA, H) = A(H) for any H G c. We transfer the scalar 
product { , ) to a* and extend it to a Ç-bilinear form on aç The Fourier transforln 
of a smooth compactly supported flmction f : K  Ç is bv definition the function 
- a x K/M  Ç given by 
(2.2) ff(A, b) =  f(z)e(-i+P)A('b}dz 
where A(z, b) is an a-valued analogue of the usual scalar product on Euclidean 
spaces (see [12, Chapter III]). For any g  G md k  OE one has by definition 
A(g, kM) = -It(g-lk). If f is -invariant (i.e., f(k. z) = f(z) for any k e  
d z  N), then  does not depend on b. The forlnula (2.2) becomes 
f(, b)= [ f(gN)ç_(g)dg 

(2.4) çx(g) = [ e(i)+P)A(gE'b)db" 
/W 
A central result in the theory of convolution operators on senfisimple Lie groups 
is the Kunze-Stein phenomenon, which states that if p  [1, 2), f  L(G) d 
K  LP(G), then 
(2.5) Il/* KllL(G)  CpI[fIIL(G)"K"L,(G)  
This inequality was proved by Kunze and Stein [15] in the case when the group  
is gE(2, N) (and, later on, for other particular groups) and by Cowling [7] in the 
general case. In [13] Herz noticed that the inequality (2.5) tan be sharpened (and 
its proof greatly simplified) if the kernel K is K-bi-invariant, i.e., K(kgk2) = K(g) 
for y kl, k2 e E md g e G. Let 111  KIL»(G ) denote the norm of the convolution 
operator defined by the kernel K and for any p  [1, oe] let pp = 12/p - l[p. One 
has the following criterion due to Herz [13]: 
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Proposition 2.1. If 1 <_ p <_ oe and K is a IK-bi-invariant kernel on G, then 
(2.6) III * lçlllL(C,) < C ] 
It is easv to see that if 1  p < 2 and H  n + (the elosure of the cone n+), then 
ç_w(exp H)  Cpe -p(u)/p', where t/= pi(p- 1) is the conjugate exponent of p. 
Thus (2.6) becollWS 
J 

3. A TRANSFEREN('E TttEOREM 

I,et 4) " n -, C i)e a fimction Oll 0. SUl)l)orted in n +. Let ? = dira n denote the rank 
()f the group . Asilme that (he filn('tiol  sa(isfies the following bsic sumption: 
there exist two COllStant.s A and c0  0 su('h that 
(3.1) Iç(Hi) - (tt2)1  .4(1 + IHII) -(e+°) 
for any Il1, ti2  n + with the t)roperty that IHI -- II21  1. 
Let p i)e a fixed expolwn in the interval (1.2) and let hp, e " G  C be the 
-l)i-invariant kernel on G giv('n i)y 
(3.2) Kp,¢(L'l(exp H)L'2) = e-p(H)/P¢(H) 
for any tt G a + and kt, k2 G . As iu the previous section, let 
denote the norm of the convolution operator wflh the kernel Iç,,o on L»() and 
III * 111,¢«) th ,n, f the covolutio operator with the kerel  o L"(n). Out 
first main theorem is the following: 
Theorem 3.1. Assume that  satises (3.1), p  (1,2). ad AÇ,o is dened as m 
(3.2). Then there ezists a constant C, dependin9 only on p and the 9rop G sch 
that 
The constant . depends ozdy on  and the constants A and Co in (3.1). 
Remark 1. This theorem is sharper than the classical transference principle of 
Çoifman and Weiss [6, Theorem 8.7] because the factor that nm.kes the transition 
between he kernels  and Içvo is e 2p(H)/p. This transition factor is (H) in [6, 
Theorem 8.7], which is proportional to e 2p(H) if H is hot close to the walls of 
the yl «hamber n +. See also [8. Theorem 4.1] and [9] for other transference 
principles. 
As an application, we have a new variant of Herz's majorizing principle. As- 
sume that the function ç is supported in the cone a + and satisfies the differential 
inequalities 
(3.3) I()1  A'(1 + I1) -- 
for any H  a + and  = 0 or  = 1, where O(H) denotes any partial derivative 
of ç of order j. Assume also that the function  satisfies the cancellation condition 
(3.4) [fO(H)dHA'.N 
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for any N > 0. hl this case the functioli çb is a kernel of the Calderon-Zygmund 
type on a. Inde('d. the conditions (3.3) and (3.4) guarantee the fact that the kernel 
¢ defines a bounded operator on LU(a) for any q E (1, oc) (see, for example, [20, 
Chapters I and VIl). One has the following consequence of Theorem 3.1: 
Corollary 3.2. If p  (1,2) and ¢ satifies (3.3) and (3.4), then 
The corstant Cp deperds orly on p and A'. 
Notice t hat, in general, t ho funct ion çb iii Corollary 3.2 is hot absolutely integrable 
at oc: thercfore, this corollary camlot be obtaincd as a consequence of the Kunze- 
Stein phenomenon or Hcrz majorizing principle. Easy examples, similar to the ones 
iii [14], show that the large scale cancellation condition (3.4) is crucial. \ also 
remark that both Theorcnl 3.1 and Corollary 3.2 are false if p = 2. 
The rest of this section is devoted to proving Theorem 3.1. We will follow closely 
the line of the proof of Theorem 1 iii [14]. Throughout the proof of Theorenl 3.1 
the letter C will denote miiversal constants dcpending only on the group , Cp 
will denote consta.nts depending on p and the group , and À will denote constants 
depending Ollly Oll f and the values of c0 and A iii (3.1). Elements of a will be 
denoted bv H, H1, and H.2, and elenlents of N will be denoted by n, m, and . 
Recall that for any locallv integrable IK-bi-invariant fiulction K and an)" smooth 
compactly supported function f  IN --, Cthe convolution f  K is defined by the 
formula 
I * ç(z) = £ I(-o)Iç(h - 
where 0 = G/IK is the origin of the synnuetric space IN. Notice that 
,,,* Içp,p,,,LV(Z) = sup I££ f(h'O)Içp,4)(h-lh')9(h"O)dhdh' I , 
where the supremum is taken over smooth compactly supported functions f, 9 " 
N --, C, and p' = p/(p - 1) is the conjugale exponent of p. 
As iii the proof of Theorenl 1 in [14], the main idea is to estinaate integrals iu 
Iwasawa coordinates. We identify the group G with N x a x IK using the Iwasawa 
deconlposition G = (exp a)IK. Let dp be the measure on N x  corresponding to 
this decomposition, i.e., dp = e'O(mddH. Thus il suffices to prove that for any 
smooth conlpactlv supported functions f, g  N x a --, Cone has 
[Ip,+(f,g)l < Cp(lll * çllL,(« ÷ À)l[f[[L(S×«a.>ll9 

(3.5) 
where 

(3.6) 

× g(, H.2)¢2P(H+H)dHldH2dd. 

By definition 5H( ) = (exp H)V(exp-H) for any H  « and V  N. Il is clear 
that H is a dilat.ion of the group . In order to estimate II,e(f,ç)l we need to 
understmld the connection between thc Iwasawa decomposition and the Cartan 
decomposition of the group G. Let +« denote the interior of the COlle {H Ç la  
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(H, H') > 0 for anv tt' E «+}. It is well knowll that. «+ C +« (see [10 Lemma 
35]). For any H E «+ let 
(H) = miu (H). 
+ 
Lemma 3.3. If ç  N and H  a+, then 
(3.7) [v( )] = n + ri(v)+ n(v, ) 
uhere H(, H)  + and there exists a constanl Co such that 
(3.) e 2p(n'(v'n)) - 1 g (e-2()e C°p((v)). 
P'oof of Lemna 3.3. This lcmma is a. quantitative version of Proposition 4.24 in 
[12 Chai)ter 1[]. R(call that g+ and H(g) dcnote thc «-components of the element 
g  G in the Çartan dccomposition and thc Iwawa dcconlposition of the group 
. It i well known that g+ - H(g)  +« for any g  G ([10, Lenmm 35]); therefore 
H'(, H)  +«. Thc estimatc (3.8) is a consequcnce of the proof of Proposition 4.24 
in [12, Cai)ter III. With thc notation in this proposition, let n be an irreducible 
rel)rescutali(»n of G on a finite-dimensional vcctor space I , A the highest weight of 
n. ami u the restriction of A to . Let 
-= ç. ç= i, + ii 
be the dccomposition of I  into weight subspaces. Fix v,..., v and w,..., w, the 
corresponding orthogonal i)is consisting of restricted weight vectors. It follows 
immediatelv from one of the equations in Proposition 4.24 in [12, Chapter III that 
(('(')) - 1)   II(v),jlld "-»)()(()). 
One clearly h 
II(v)ll  '). 
Therefore. it suces to prove that for any   N, 
(8.9) p(V  )  C + C0p(n(V)). 
Notice first that this incquality follows from the explicit formulae in [12, Chapter 
II, Theoenl 6.1] if the group G bas real rank one. One cau also take C0 = 1 in this 
case. Iu the general ce, let X, X2 .... , X be a basis of the Lie algebra  with 
t he followiug properties: 
(i) for any 1  j  r, the vector Xj belongs to a foot space 0- for some root 
 Z+; 
(ii) if ff denotes the linear span of the vectors X .... , X then ff is an ideal of 
ffforanv ljr. 
The map "   N, 
(,...,x) = (epx.)--- (xpxX) 
gives a pa.rametrization of the group N. For a.ny  = ç(x ..... x)  N let 
Ilvll =  I1- 
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A similar "I1O1"111'" ftnlction on I was used in [16]. It follows frolll [16, Lemma 5.3] 
that there exists a constant ci > 0 such that 
-2p(t-/(v)) <_ C(2 + IIvll) -«1 

Thus 
(3.10) 
We will now show that 
(3.11) 

log(2 + Ilvll) < <' +C,p(H()). 

Let d( , ) denote the distance function ou the s.vmmetric spacc E induced by the 
Killing form on 9. One bas I+1 = d(, e). Thereforc, if  = (x .... , x), then 
j=l 
It follows flore the rank-one rcduction mçthod and the cxplicit rank-onc formulae 
in [12. Chaptcr II. Thcorem 6.1] that 
d((exp«iXi)oE, eoE)  Clog(2 + Il)- 
Thus (3.11) follows from thcse last two estilnatcs and (3.9) follows from (3.10), 
(3.11), and the observation that p() 
 will use this lemma to estimate the function (, H)  Içv,(V(expH)) for 
any    and H  . Let P() = e-PH()); it is well known (see [10, Lemma 45]) 
that for any e > 0, 
(3.12)  p()(l+e) = 
(i) If H  + and   N, tben 
Ilçp,4(V(exp H))I 
H G o + and  G N, then 
Içp,4(V(ex p H) ) = e-2P(H)/pç( H)P(V) 2/p + Ep,4(V, H) 

Lemma 3.4. 
(3.13) 
(ii) If 
(3.14) 
where 
(3.15) 

IG,(,H)I < c. À¢-2p(m/(1 + IHI)-(t-+«°)t(v) / 
(1 + p(H)) + 
Thc constant C is the saine as in Lernma 3.3. 

Proof of Lemma 3.4. Notice first that 
lira çb(H) = 0. 
This follows immediately from (3.1) and the fact that III * &IllL(«) is finite. Thus 
it follows ri'oto (3.1) that 
(3.16) Ib(H)[ <_ .(1 + [H[) -(e-'+«) 
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for any H E a +. To t)rove t)art (i) of the proposition notice that 
lçp,¢(ii(exp H)) = e-29(H+H()+H'('H))/Pdp(H + H(V) + H'(Y, H)). 
Recall tha) H(Y) and H'(ç,H) belong to +a ([10, Lcmma 43]). Also, it is well 
known that p(+a) > 0, i.e., H9 C a +. Tlms (3.13) follows esily from (3.16). 
To prove pari (il) notice lhat it follows from Lemma 3.3 that the difference 
t)etween K,o(Y(expH)) and e-29(H)/v(H)P(Y)2/ is equal to 
12()/PP() 2/p [--2(H'('))/P(H + II() + H'(D,H)) - (H)] . 
Since H() and H(F, H) bclong to +n, one h 
)H(v) + H'(V,H)I  @(H(V)) + eH'(V.H)). 
One tan now combine th(, estimates (3.8) and (3.16) and the bic inequality (3 1) 
to show that 
p,o(. H)[  O- 
In addition, it folh)ws rioto (3.13) and (3.16) that 
Ip,o(, H)[  À«-°()/(1 + [H)-(e-+«)P() / 
and the estimatc (3.15) follows froIn the lt two inequalities. 
Xç% return to the proof of Theorem 3.1. Let k be the characteristic function of 
the set a+; we decompose the integral I,o( f. g) in (3.6)  I,o( f, g) = I,o( f, g) + 
I,o(f.g) + I,o(f,g), where 
x g(, H)e9(n'+n)(l - (Ha - H))dHdH, 
 g(, H2)e 2p(H'+Ha) k(H2 - H)dHdH, 
alld 
(.) 
We record an elementary fact that will be used several rimes in the rest of the 

paper. 
Lemma 3.5. 

One bas 
(3.18) _(1 + [HI)-(e-I+«°)e-«'(H)dH <_ C«o,«  
for any eo, ci > O. 
Lemma 3.6. One bas 
(3.19) I' . 
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Proof of Lemma 3.6. Let 
G(H2) = [flg(, Hu)lP'a] /p' 
By HSlder's inequality, the absolute value of I,¢(f,g) is dominated by 
Recall that the nmp   2 = 5n() is a dilation of  with  = e-2P(H) 1. 
In addition, it is wcll known that the Abel transform 
takes suitable OE-bi-invariant fimctions k to lI'-ilvariant functions on a. For any 
regular element H  a let H + 6 a + I)e its representative in the positive yl 
chamber, i.e., H + 6 (II"  H)  a +. Thus, if H  a is regular, one h 
)1 
 Cp. ÀeP(H+)«-P(H)e-2P(H+)/P(1 + [H]) -(e-l+°). 
The lt estilnate is a consequence of (3.12) and (3.13). It follows after several 
simplifications that the absohtte value of the integral in (3.20) is dominated by 
The change of variable H = H + H shows that this integral is equal to 
Cp. I (I(F(HI)e2p(H')/P)(G(H + H)e2p(H+H)/p')dH ) 
x (1 - (H))(1 + [H[)-(e-+)e-P(+-)(2/P-)dH. 
By HSlder's inequality, the inner integral is dominated by 
(I[FI(HI)[Pe2p(H)dH)/P(ZG(H2)[P'e(H)dH)/P', 
which is equal to 
Thus one only needs to estimate 
(.1) (- ())(1 + 
Notice that if H  a , then p(H + - H)  w(H), where w(H) h the saine meaning 
 in Lemma 3.3. Since p < 2. it follows from Lelnma 3.5 that the integral in (3.21) 
is dominated by Cp   and the lemma follows. 
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Lemnla 3.7. O-ne /tas 
ProofofLem-ma 3.7. By [t6hler's inequality, the ahsolute value of the integral 
I,¢(f, g) is donfinated bv 
(.) 
Notice first t hat 
One can use (3.15) to estimatc the integral of Ep,ç. [t follows from (3.12) that 
./ P()/P(1 + p(H(9))  Cp. 
Also 
 P(9)/P rein( l, e-(H)eC°P(H(v)) )dU 
 e-(H) (H())w(H)/Co p()2/p + (H())(H)/Co P(5)/P 
 Ce -w(H) + e-(1/2+I/p)p(H())e-(1/p-1/2)w(H)/Co 

(3.2) 
where 

<_ Cpe-c,w( H) 
where cp = (1/p- 1/2)/Co > 0. It follows froln the la.st two estilnates and (3.15) 
that if H E a +, then 

q,(H) = (1 + IHI) -<e+ço) + (1 + IHI)-(e-+«°)e -.(H). 
Let, (H) = 0 if H  a +. It follows from (3.18) that   L(a), i.e., 
One substitutes the estimate (3.24) in (3.23). The change of variable H2 = H + H 
shows that the absolute value of I,¢(f,g) is donfinated by 
C'. I (I(F(H1)ep(H1)/)(GI(HI + H)ep(I+H)/F)dH,) ,(H)(H)dH. 
The lcimna now follows from this last estimate, H61der's inequality and (3.25). 
Lemnm 3.8. One bas 
(3.26) 
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Proof of Lemma 3.8. The change of variable H2 = H1 n u H shows that Iap,¢(f, g) is 
equal to 

(3.27) 
where 

One has 
(3.28) 

Let 

£ fN a f(-'H1)C2('H1)P((-HI('-I))2/pe2p(HI>e2p(H1)/PdHld-d- 

(2(, H1) = a g(" H1 + H)e2°(HI+H)/P'¢(H)dH. 

(fa )l/p' (fa 
[G2(,H1)[P'dH1 <-Iii * dP[[ILv(.) 

(f 
C3(H1) = 
It follows by H61der's inequality and (3.12) that the absolute value of the integral 
(3.27) is dominated by 
aF1 (H])e 2p(HI)/pdH1, 
which, again by HSlder's inequality, is doninated by 
' / 
(IÇ3(H1)P'dHI (IFI(H1)Pe2P(HI)dHI) 
The estimate (3.26) follows. 

The main estimate (3.5) is a consequence of (3.19), (3.22) and (3.26). 

4. L p FOURIER MULTIPLIERS 

Recall that the Fourier transform on the symmetric space 1 associates to any 
smooth compactly supported function f on 1 a function f" az x K/NI  C. By 
the Plancherel theorem, any bounded IV-invariant multiplier m : a* - C defines 
a bounded operator Tr on L(1¢) given by Tf(),, b) = rn(A)](A, b). Assume that 
p (1,2) U (2, oe) and let pv = [2/p- l[p. Let 
Tp = a* + ico(ll"- pp), 
where co(W- pv) denotes the interior of the convex hull of the set of points {w. pv : 
w  W}. 
Assume that one has a multiplier m : a*  Cthat extends to a bounded W- 
invariant holomorphic function in the interior of the tube Tp. Assume, in addition, 
that m satisfies the differential inequalities 
(4.1) 
for any j = 0,1,... and A G Tp. Here d(A) denotes the distance between the 
point   2r and the set i(a* \ co(W-pp)). More precisely, if  = r + i, then 
d(£) = (Irai  + d(,a*\ co(II'-pp)))/2. As before, c.m(A) denotes any partial 
derivative of m of order j. 
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Theoreln 4.1. ff t' E (1, 2) (_J (2, oe) and m satisfles the d(fferentiai inequalities 
(4.1), tben the operator T,n e.rh",d.s fo a bomtded operator on LP(). 
An ol)crat.or T, dcnc(l 1)y a multiplier m that satisfies (4.1) can be thought of 
as a singular integral Ol)('rator on the svlnlnetric sl)ace X. The rest of this section 
will be dovoted to proving Thcorcm 4.1. The lctter Cwill denote constants that 
mav depend on the group G and finitely many of the constants Aj in (4.1), and C v 
will denoe constants that mav also depend on p. 
One can assume hat p  (1, 2). In or(l(,r to insure the convergence ofthe integrals 
throughout this section we will also assume that the nmltil)lier re(A) is premultiplied 
with a factor of the form e -6('), where 0 <   1. Our estimates will be uniform 
in 5: once one proves suital)le uniform est imates, standard limiting arguments allow 
(me to l)ass to thc gen(al thcorcm. The multiplier m is holomorphic in the interior 
of lhe tul»(. ; thcrcfi)re we can assume that for any ç  co(II', pp) the function 
q  m(q + i) is a Schwartz fimctkm on 
Let lç be thc K-l»i-invariant kern('l of tire Ol)('ralor T. Bv thc inversion formula 

ont has 

/ ,,,(A)ç(exp H)Ic(A)I-2dA 
(4.2) K(exp H) = C, 
for any H E a +. Thc sphcrical fimctions ç, are defined in (2.4) and c is the 
Harish-Çhandra fimction. The idea of the proof is the following. Using smooth 
cutoff functions we will break up this kernel into thrce parts. The first part is 
supported near the origin of the group  (i.e., in the set {E(exp H)E" IHI 
The analysis of this local part of the kernel is contained in [1, Section 4]. The 
second part. of the kernel is supported along the walls of the positive Xyl chamber, 
i.e., in the set {E(expH)E  H  a + and (H)  2}. One can use the Herz 
majorizing principle (see Proposition 2.1 and (2.7)) together with the main idea in 
[1. Section 2] to deal with this part of the kernel. Finally, the main part of the 
kernel is supported away from the walls of the positive Xyl chamber, i.e., in the 
set {E(expH)E  IH I  1 aud (H)  1}. The Herz majorizing principle fails to 
prove boundedness on LP(Z) of the operator defined bv this part of the kernel if 
the multiplier n h a singularity at the point ipp.  use Theorem 3.1 and the 
Harish-Çhandra expansion of the spherical fimctions ç to control the norm of the 
operator defined by this part. of the kernel. 
Let   a +  [0, 1] be a smooth cutoff function such that ,(H) = 0 if IHI  1 
and I(H) = 1 if IH  2. Also, let   a +  [0, 1] be a smooth cutoff function 
such that (H) = 0 if (H)  1 and (H) = 1 if (H)  2. The kernel K in 
(4.2) can be written as K = K + K + K3, where 
Içl (exp H) = (1 - ¢1 (H))h'(exp H), 
 (exp H) = 1 (H)( 1 - 2(H) )K (exp H), 

and 

K3(exp H) = ¢1 ( H)p2(H)K(exp H) 
for any H G a +. The kernels K, K2 and ik'3 are of course ]K-bi-invariant kernels 
on G. It follows easily from [1, Corollary 17] that the kernel h defines a bounded 
operator on L'E), i.e.. 
(4.3) 
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It remains to prove similar estimates on the norms of the operators defined by the 
kernels Iç2 and K3. 
Lemma 4.2. One bas 
Proof of Lemma 4.. This proof is based on the main idea in Anker's paper [1]. By 
Proposition 2.1 it suffices to prove that 
(4.4) 
As in [1], for any r k 0 let 
lç = {H  a " (u,. p)(H) < ]p]r for any ,t,  II} 

alld 

Ur = K(exp Vr)K. 
\Ve will prove that for anv integer r _> 0 one bas 
(4.5) [ IIç2(g)ldg < Cveoelplr/p'(1 + r) -a/oe. 
du 
+ku 
Notice that e -2(u+)/F  e -21I/F if g  U+  U. Thus (4.4) follows froln (4.5) 
by SUmlnation over r. 
X need the following estimate on the mesure of the set (tÇ+ 
(4.6) ](Ur+l /lr)  supp K2I  C(1 + r)e-2e 2M. 
Notice that the power of (1 + r) in (4.6) is g - 2 rather than  - 1 (compare with 
the estilnate in [1, Lemma 6]). This is because of the "'small" support of the kernel 
K2, and this gain is essential for the proof of the lemma. To prove (4.6) notice t.hat 
for any H  (I5+1  I.)  a + one bas 
(H)  e 2p(H)  Ce 211, 
where (H) is the density measure defined in (2.1). In addition, one can show eily 
that the measure in a of the set (Vr+l  1;)  a +  (supp (1 -- 2)) is dominated by 
C(1 + r) -2. The estimate (4.6) follows. 
By H61der's inequality and (4.6), one has 
+U 
for any integer r k 0. Thus it suces to prove that 
(4.7) IIIçllL=(U+,W)  Ce-'( 1 + 
Let  denot.e t.he Fourier transform on the symmetric spe  defined in (2.3). 
It is well known that  extends to an isonorphism between S(//K) (the L 2 
Schwartz space of K-bi-invariant functions on ) and S(a*) W (the subspace of 
IV-invariant functions in the Schwartz space S(a*)). See [1, Section 1] for the 
definition of S(//K) and references. For any f  C(G//K), let (f) denote the 
Abel transform 
e (") [ I((p H))a. 
A(f)(H) 
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Thc Abel transform extends to an isomori)hism i)ctwecn S(G//IK) and S(a)  (see 
[1, Proposition 3]). Finally, let. Y (h'lmtc the Euclideal Fouricr transfornl 

Y(g)(A) ----/ g(H)e-'(1)dH. 

which is an isolnorphism between S(a) u and S(a*) u . In addition, for any f E 
S(Ç//K) one has 
7-t(f) = .Y(A(f)). 
Let L = Y-l(m) = A(K). Assume that r _> 2 in (4.7) (only simple modifications 
are needed if r = (I or r = 1) and let gr " a -- [0, 1] be the ll'-invariant smooth cutoff 
funçtions defined in [1, Sect.ion 2]. These fimctions are supported in the conlplement 
of I'_ and are equal to 1 in the comt)lelnent of çr. The key observation in [1, 
Section 2] is that 
K(expH) = A-(L.gL.)(H) 
for any H outside Ur. This is a simple consequcnce of a support property of the 
Abol trm,sform (see [1. Proposition 4]). Thus, by tl,e Plancherel theorem, one has 
< C[[2(L  tr)[[=(...l¢(),-=a ). 

Olle also ha.s the estilnate 

Ic()l - _ C1 + ]AI2) 2b 

for any A E a*, where b is a fixed positive integer. By the Euclidean Plancherel 
theorem 
II-r(L )[l=(«,l()l-=e) < Cil-r( L" ,-)(,)(1 + 
b 
j=o 

where A« is the Laplace-Beltrami operator on n. Thus it suflïces to prove that for 
any integer j E {0.1, .... b} one bas 

(4.8) 

IIA{(L " ffr)llL=«)  Cve-l°l"(1 + r) -(+1)/2- 

Recall that L is the inverse Euclidean Fourier transform of the nmltiplier m; 
therefore, 

(4.9) 

L(H) = C . m(A)ei'(U)dX 

Assume that H E a + and p(H) >_ 1. Since the multiplier m is holomorphic, one 
can shift the integration in (4.9) to the space i(1 - )p» + a*, where oep(H) = 1. 
One has 
Cpe-Pv(H) Ja [- ?rt(A + i(1 - e)pp)ei)(H)d,. 
L(H) 
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Ve integrate by parts N t.iules in  and use (4.1). Notice also that if H E +a, then 
p(H) . ]H]. Thcreforc oue has 
, ]m( + i(1 - )pp)d 
(a.10) 
if 2N >  + 1 and H  a (a l _). The kernel L is l I':invm-iant and the measure 
(in ) of the set IÇ,  l,_ is donfinated by (1 + r') e- for any r'  1. The estimate 
(4.8) follows for j = 0. One can easily reI>eat thc argmnent for any j  b and 
Lemma 4.2 follows. 
It remains to prove the following estimate: 
Lemma 4.3. Oe bas 
Proof of Lemma 4.3.  will use thc Harish-Chandra expansion of the spherical 
fimctions ç(H). Let 
and let 
B = { e «* - (H) > -0 for anv   Z +} 
be an open ucighl)orhood of a. The coustant 0 will Le chosen small and strictly 
poeitive. Let Q be thc positive lattice generated by the simple roots   Z +. If 
H  a + and A  *. then one has the aLsolutely converging expansion 
(.) ¢()1-() = « 
qE2Q wEI 
The coecient F0 is equal to 1: the other coecients çq are rational flmctions in 
A and exteud to holomorphic fuuctions in the tube «* + lB if the constant g0 is 
chosen sml enough. Moreover, there exist constants Ç and d such that 
(4.) 
for any £ G «* +iB and q G Q.  substitute the expansion (4.11) into the inversion 
formula (4.2) and notice that the integrand is iV-invm'iant. One has 
(4.a) 
= 
qQ 
The main terre in the sure al)ove corresponds to q  0. Ve use the Herz 
majorizing principle to estima.te the L  norms of the operators induced bv the 
error terres in (4.13). For any q G Q let 
(4.14) Iç(exp H)  C(H):(H)e-(u)e -qu) . m(£)c(-£)-Fq£)e(U)dX 
In general (i.e., if g  2) the functions Fq do hot satisfy favorable symbol-type 
estimates on «* except for the uniform inequalities (4.12). However, one can use 
the fact that they are holomorphic functions in the tube «* + lB to estinmte the 
absolute value of K(exp H). This idea was used in a recent work of Anker and Ji 
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[3]. 
ctEE + and let 

Sec also [2] for similar estimates. Let E ++ denote thc set of indivisible roots 

(4.15) P(A) -- H (1 - i(c,A>) k«, 

whcre k« arc certain large integers. Thc flmction c can be computed explicitly 
(see, for example, [2, Section 2]). This explicit formula shows that the function 
A  c(-A) - is holomorphic in the tui)e a* + lB if e0 is small enough. In addition, 
there are positive constants ka' such that 
c(-A)-' I  C H I-i<o,A)[ ' 
aEE++ 
for any A  a* +lB. Thus one can fix thc constants k« in (4.15) so that the function 
c(-&)-/P(A) bclongs to the spacc H(T) -see [19, Chapter III] for thc theory 
of H  spaccs on tubes. By (4.12) the flmcfion A  F¢(A)c(-A)-/P(A) belongs to 
the space H(TB) and 
.. I'(, + )(-, - i) -/P(, + i)l  d, 5 C( + Il)  
for any   B. One h 
,(xp ) = c()()«-(m¢-«y -' (  ()(-)-'c()) 
= C(H)e-(le-¢()Y- (m. P)  
where @ = g,@ and le(A) = ç¢(A)c(-A)-/P(). By [19, Theorem 3.1, Section 
III], the flmction Y-(f¢) is supported in the cone -+a and satisfies 
/ I-()(H)IH  C(1 + Iql) . 
On thc other hand, - (m- P) is a certain derivative (in H) of the flmction L = 
-(m). An argument similar to the one at the end of Lemma 4.2 (see (4.10)) 
shows that 
lY-' (m. P)()l  c,e-()(  + ,())- 
for any H  +a with the property that IH] k 1. Combining the lt three equations 
and HSlder's inequality one h 
(4.) 
 C@(H)e-()e-¢() [ - (m. P)(H - H')[[-(f¢)(H')]dH ' 
 Ç,(1 + Iql)(H)e-(")e -«" ly- (m. p)(H + H')ldn' 

for any H E a +. The Herz majorizing principle applies if [q[ _> 1. Recall that the 
support of the cutoff function !b is included in the set {H G a +  w(H) _> 1}. Bv 
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(2.7) and Lemma 3.5 one has 
III * tçlllLp(x) < Cp + [tç(exp H)[5(H)e-2P(H)/P'dH 
_< Cp(1 + ]ql)de -(Iql-l) .+(1 + IH])-ee-w(tt)dH 
It remains to deal with the kerncl tç °. We need one inore redu«tion belote we 
can apply the results in the previous section. For any H  «+ let 
(exp H) = CI (H)e -p(I m()e(-)-e(Id 
(compare with (4.14)) and extend K to a N-bi-invariant function on . Notice 
that the fimctiou 
a+ " (H) N 2} and satisfies an estimate similar to (4.16) for q = 0. By (2.7) and 
Lemma a.5 oue has 
It remains to prove a similar inequality for the kernel K. This will be a conse- 
quence of Corollary 3.2. For any H  a + let 
(4.17) 
and ç(H) = 0 if H  «+. % bave to prove that ç satisfies conditions (3.3) and 
(3.4) in the previous section. Notice that for anv H  a + one h 
ç(H) = 
= CI(H ) [ (ePP(H-H')-I(HI" P)(H-H'))(ePP(H')-l(fo)(Ht))dH'. 
Recall also that -(f0) is an L  fimctions supported in the set -+a. Therefore 
the estimate (3.3) is an easy consequence of the following estinmte: 
(4.18) 
for any H  +a with the property that p(H) k 1 and for j = 0 or j = 1. Let 
The proof of (4.18) for j = 0 is identical to the proof of (4.10) in Lemma 4.2. We 
will now prove (4.18) with j = 1 and H = H0  +a awv from the origin. Let 
« = p(H0) - and notice that 
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by a sllift of integration t» the spa«e a* + i(1 - e)p v. 
One has 

O (ito) 

Fix Hl  a with [Ht[ = 1. 

<- CI#H°)-' /i" m(A + i(I -e)pp)P(A + i(1 
+C[[.. m(A+i(1 

lel CI lu' lhe degree f lhe polyn«mtial P. The estilnate (4.18) for j = 1 follows by 
integra.ti«m iy parts as in (4.10). ()lC only u««ds to notice chat, a.s a consequence 
f (4.1), 
[A. (A ---, ,,,(A + i(1 -e)p,)P(A + i(I -e)p,)[ < CN(e + IA{)-9-N(1 + lAI) < 

«IIRI 

[A.(A --+ n(A + i(1 - e)pv)P(A + i(1 - e)pv)A ( 
< Cx(«÷IAI)I-2N(1 ÷ lAI) < 
t»r anv inleger A" OE I), whero, as I»('fl»re, a" (h'noles lhe l,al)lace- Beltrami operator 
Thc l)roof of lhc canccllation ('(m(lilion (3..1) is somewlmt delicate, mainly be- 
cause thc support of thc fim('li(m ç is restricted to thc conc a +.  proceed  in 
[14]. L(, p   + [0.1] !,(, a 
and (,(tual lo 1 in lhe inl('rval [2,,). I,et Z+++ 1)c the set of simple roots in 
Cille] hH" ally ti  a let 
 (H)= H ,(.(H)). 
Noice that we can relace the 
and notice tire{ he rç.i,, 
and satises the estimate (3.3) Rn" j = 0. Thus @-@ is an E 1 function. It remains 
to prove that the fimction çl salises the cancellation condition (3.4). Recall that 
br any E «. hus lhc canccllalion condilion (a.4) is equiwleut o 
(4.19) ]J2 0I(H)7(eH)  C 
for anv e  (0.1/2], where 7 " «  [0.1] is a smooth cutoff fimction supported in 
the ball {H  «" [H[  2} and equal to 1 in the ball {H  «" [H[  1}. Let 
T = -1() be a distribution on a*. Bv the saine argmnent  in [14. Section 4], 
it suces to prove that 
(.20) I T (t  5('t- ,t0 + (1 - 
for anv q0  a* and any ¢  (0.1], where Ç is the saine constant as before and 
3(A) = m(A)c(-A) -. For anv   E +++ let T : -I(H  a(a(H))). Oue h 
for anv Schwartz function f  «  C, where ai ..... a, are the positive simple roots. 
The estimate (4.20) will be a consequence of the following lemma: 
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Lemma 4.4. Assume that D1 is an open set in the Euclidean space ', u E ' is 
a fixed vector wzth lui = 1 and D2 is an open set with the property that D2 + tu C Dt 
for any t  [0, 50], 50 > O. Let Tu be th« inverse Fouri«r transform of th« function 
v  #((u,v)) (in the sense of distributions). Assume that f is a holomorphic 
function in the tube ' + iDt with the property that 

]f(x ÷ iV)]  .4(1 

for any x  ' and y  Dt. ht addition, assume that the function x --* f(x + iy) 
is a Schwartz function on n for an, y y G Dt. Then the functzon 

bas the property that 

f * T,(x + iy) =- T,(v  f(x - v + iy)) 

If * Tu(x + iy)l <_ C6 o  A(1 + Ixl) 61 

for any x G n and y G D2. In addition, the function x -- f , T,,(x + iy) is a 
Schwartz function on n for any y G D2. The constant C5o depends only on 5o and 
the cutoff function p. 

The estinmte (4.20) follows from this lclnnla by a simple inductive arglllllent. 
One starts with f(A) = (-) and DI = -«o(II"- pp) and applies Lemllm 4.4 to 
the vectors  = H ..... H. Olm only needs to check that there exists a Slnall 
constant (0 -=- 6o(p) with the property that 
(4.21) (1 - e)pv - (tic +... + tette)  co(II"- pv) 
for any tt,..., te  [0,60]. Recall that e G (0, 1/2]. We need the following well- 
known fact about convex hulls (sec [11, Chapter IV]): if H E a + and C(H) is the 
closure of the orbit {w  H  w G W}, then 
C(H)  a + = (H - +a) Va +. 
We tan pass this to the space a* attd apply it to the vector p. One has 
C(p) ç a+ = (p - +-K) ç a+ 
where C(p) is the closure of the set co(lI"  p) and, as before, + and +* are the 
cones corresponding to + and +«, respectively. Notice also that cq,..., ce G +*. 
Thus one has 
p- (tt +... + t,e«)  c(p) 
for any t ..... te  [0, ci and (4.21) follows. [] 

Proof of Lemma .. This is a consequence of the one-dimensional estilnate proved 
in [14, Section 4]. Assume that u = (1,0 ..... 0). The distribution Tu is the in- 
verse Fourier transform of the function v  #(v). The saine argument as in [14, 
Section 4] shows that 
f oe(t)(f(t,O ..... O)- f(O)« -t) dt 
T(f) 
+ 
Cf(O) 
for any Schwartz function f, where u is the inverse Fourier transforln of the fimction 
#. Thus 
(4.22) f. T,(x + iy) = f u(t)(f(x - tu + iY)_it- f(x + iy)e -) dt + Cf(x + iy) 
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wh(u'e f is the fun('titm in I,emma 4.4 and y 6 D2. The flnlction u, the inverse 
Fourier tralsform of the Slnooth ('olnpactly supported flmction p', extends to a 
holomorphic flmction in the plane. We shift the integration in (4.22) to the line 
-lb0 + , whcre 0 is the constant in Lemma 4.4. Notice also that the function 
t  v(t - "io) is a S('hwartz flmction on  and the lemnm follows.  
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WEST'S PROBLEM ON EQUIVARIANT HYPERSPACES 
AND BANACH-MAZUR COMPACTA 

SER(;EY ANTONYAN 

ABSTRACT. Let G be a compact Lie group. X a metric G-space, and exp .\ the 
hyperspace of ail nonempty compact subsets of .\ endowed with the Hausdorff 
metric topology and with the induced action of G. We prove that the following 
three assertions are equivalent: (a) X is locally continuum-connected (resp.. 
connected and locally continuum-connected); (b) expX is a G-ANR (resp., a 
G-AR); (c) (expX)/G is an ANR (resp., an Al:(). This is applied to show that 
(expG)/G is an ANR (resp., an AR) for each compact (resp., connected) Lie 
group G. If G is a finite group, then (expX)/G is a Hilbert cube whenever 
X is a nondegenerate Peano continuum. Let L(n) be the hyperspace of all 
centrally symmctric, compact, convex bodies A C I n, n >_ 2, for which the 
ordinary Euclidcan unit ball is the ellipsoid of minimal volume containing A, 
and let Lo(n) be he complement of the unique O(n)-fixed point in L(n). Ve 
prove that: (1) for each closed subgroup H C O(n), Lo(n)/H is a Hilbert cube 
manifold; (2) for each closed subgroup K C O(n) acting non-transitively on 
S n-1 , the K-orbit space L(n)/K and the b,'-fixed point set L(n)[K] are Hilbert 
cubes. As an application we establish llew topological models for tha Banach- 
Mazur compa«ta L(n)/O(n) and prove that L0(n) and (exp S n-l) \ {S n- 1 } 
have the saine O(n)-homotopy type. 

1. INTRODUCTION 

In 1976 .J.E. Wcst [33] asked the following question: Let G be a compact, 
connected Lie group. Is Ihe orbit space (expG)/G an absolute retract, and if so, is 
it always homeomorphic to the Hilbcrt cube? In a more general form this problem 
appeared also in [34, Problem 1022]. 
These questions bave remained open except when G = S 1, the circle group, 
where the answers are "Yes" and "No", respectively. Toruficzyk and West proved 
in [29] that the orbit space (exp 0 Si)/S  is an Eilenberg-MacLane space K(Q, 2), 
where S  is the circle group and Q stands for the rationals. 
Recall that if G is a compact group and X a metrizable G-space, then exp X 
denotes the hyperspace of all non-void compact subsets of X, equipped with the 
Hausdorff metric topology and with the induced action of G. \Ve use exp 0 X for 
the complement (exp X) \ {X}. 
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In t he first part of the prcs('nt paper we give a positive answer to the first question 
(ff Wcst's t)r(i)lem as a corollary to the filh)wing general result. 
Theorem 1.1. Let G be a compact Lie 9rottp and X a metrizable G-space. Then 
ttte followirt9 are equivalcnt: 
(1) X is locally co,tinuum-counected (resp., connected and locally continuum- 
co,,ected), 
(2) ext)X is a G-ANR (resp., a G-AR), 
(3) The orbit spo«e (ext)X)/G is a,t .4NR (re.sp.. an AR). 
The proof of this theorem is given in Section 3. 
Since each compact Lie group G is locally path-connccted, Theorem 1.1, in par- 
ticular, yicMs a. positive answer to thc first qncstion of st's Problem (Çorolly 
.S). 
Rcmark 1.2. As is shown in Section 3, the iml)lications (1)  (2)  (3) in The 
rem 1. l arc truc also fi)r mbitrary compact (hot nccessarily Lie) groups; (1)  (2) 
and (1)  (3) can 1)oregarde(l  cquivariant versions of jdyslawski's Theorem 
In Hcisey and i,st [17] it wu proved that if G is a finite group and X is a 
nondcgcnerate Pcano contimmm, thon (exp X)/G is a Hilbert cube if it is an AR. 
Conscquently, in combination with Thcorcm 1.1. this implies that (expX)/G is 
alwvs a Hilbcrt cube whcncvcr X is a nondegeneratc Pcano contimmm (Corollary 
3.9). 
The second part of this paper is devotcd to the Banach-Mur compacta. 
In his 1932 book Théorie des Opérations Lindaires, S. Banach [10] introduced, 
for each n  2, the space of isometry cluses [E] of n-dimensional Banach spaces 
cquipped with the metric 
d([E], IF]) = lninf{lITl[-IIT-'II[ T- E  F is a linear isolnorphism}. 
These spaces are now denoted by BM(tt) and cNled the Banach-Mazur compacta. 
The topology of these spaces continues to be of interest, and the following questions 
of A. Pelczyfiski wcre included in ,]. st's list of problems iii the 1990 book Open 
Problems in TopologB [34, Problem 899]: (1) Are the Banach-Mazur compacta 
BM(.) AR's? (2) Are they Hilbert cubes? 
The AR part of Pelczyfiski's problem h been solved armatively due to efforts 
of P. Fabel [16] and the author [8]. While the question of whether the Banach- 
Mazur colnpacta B;I(n) are homeomorphic to the Hilbert cube remains open for 
ail n  3, it was answered negatively for n = 2 in [9]. 
In the second part of the paper we establish some new properties of the Banach- 
Mazur compacta and consider their relation to st's problem above. It turns out 
that Pelczyfiski's problem and st's problem are of the saine nature, and both 
problems can be considered from a unified point of view. 
We recall some necessary notation first. By B(n) we denote the hyperspace of ail 
centrally sylnmetric (about the origin), compact, convex bodies in N .  consider 
the Hausdorff metric topology on B(n) aud the natural iuduced action of the full 
linear group GL(n) on it. As usual, we shall use O(n) for the orthogonal group. 
It is well known that BM(n) is holneomorphic to the orbit space B(n)/GL(n) (sec 
[34, p. 544]). 
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According to a classical thooreln of F. John [21], for an5' ,4 E B(n) there is a 
unique nfilfilnal-vohnne ellipsoid /(A) containilg A (respectively, lnaximal-vohmle 
ellipsoid j(A) contained in 4). Usually j(A) is called the John ellipsoid of A aud 
/(A) is called thc L6wncr ellipsoid of A. 
Let J(n) and L() be the O()-invariaut subsets of B(n) consisting of ail bodies 
A  B(.n) for which the ordinary Euclideau unit ball 
25} 
B'= {(.,..., ,) e a"l î +---+  
is the John ellipsoid and the L6wner ellipsoid, respectively. Solne properties of 
these O()-spaces are st.udied in [9]. It was proved in [9, Theorem 4 and emark 
1] that J() and L() are global compact O(n)-slices for the GL(n)-space B(n). It 
then follows froln a result of H. Abels [1, Lelnma 2.3] that J(n) and L(n) are O(n)- 
homeomorphic. The Banach-Mazur compactuln BM(n) is just the orbit space 
J(n)/O(n) [9. Corollary 1] or. equivalently, the orbit space L(n)/O(n) (sec [9, 
emark 11). In what follows we shall use the lnodel BM(n) = L(n)/O(n). Bv 
L0() we denote the complemel,t L(,)  {B"}. and Blo(n ) = Lo(n)/O(n ). As 
usual, we reserve the letter Q for llle Hill)ert cul»e. 
Here we prove tirer basic properlies about the Bala«h-Mazur COlnpacta BM(), 
 n >2. 
Theorem 1.3. For ay «losed subgroup H C 0(), the orbit space Lo(n)/H is a 
[0, 1)-stable Q-manifold. In parti«ular, BMo(n) is a [0, 1)-stable Q-manifold. 
Recall that a Q-manifold is said to be [0, 1)-stable if it is holneomorphic to its 
product with the half-open interval [0, 1) (sec [12. Ch. V]). 
Theorem 1.4. For each closed subgroup K C 0() actig non-tratsitively on 
S -, the K-orbit space LO)/K. as well as the h-flxed point set L()[K], is a 
Hilbert cube. In partcular, L() is a Hilbert cube. 
These theorems are proved in Section 5. However, Section 4 should also be 
considered as a part of those proofs, because the technique we develop there is 
further applied to Theorems 1.3 and 1.4. 
Remark 1.5. Below, in Lenmm 7.4. the hypothesis that K acts non-transitively on 
the sphere S "- is shown to be equivalent to the condition L0()[h'] ¢ . 
Next, in Section 6 we apply Theorem 1.3 to establish a new topological model 
for the Banach-Mazur compacta BM(n) for arbitrary t k 2. 
Namely, smne that (H), (H2),... is the sequence of all O(n)-orbit types oc- 
curring in L0(). Let COIlP(O(")/H,) denote the COII over O(n)/H, endowed with 
the quotient topology and with the translation action of O() on the levels. Let 
n(,) =  ç(g,), whr Q(g,)= (Cn(O()/g)) , 
each equipped with the diagonal O(n)-action. 
Since Col.e(O(,,)/,) e AR. i  i, it then fo,ows from a result of Xst. [32] that 
H(n) is a Hilbert cube. 
Theorem 1.6. For each closed sub9roup  E O(n), th.e o H-orbit spaces 
Lo(n)/H and Ho(n)/H are homeomorphic. In particular, th.e Banach-Mazur com- 
pacttm BM(.) is homeomorphic fo the orbit space H()/O(). 
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Bemark 1.7. In conlbination with [9. Corollary 10], Theorem 1.6 gives yet simpler 
topological models for BM(2). 
The idea tha.t Pelczyfiski's l»roblem on Bana«h-Ma.zur COmlmct.a is closely related 
to West's problem on equivarim,t hyperspaces was expressed in [9] in the following 
fornl: 
Conjecture 1.8. For each closed subgroup H C O(n). n >_ 2. the orbit spaces 
Lo(n)/H and (exp0 S'-I)/H are homeomorphic Q-manifolds. In particular, the 
Banach-Mazur compactum Bill(n) is homcomorphic fo (expSn-1)/O(n). 
Since the sphere ,b 'n- is O(n)-h, mmomorphic to the coset space O(n)/O(n- 1), 
we see that (exp.b"-)/O(n) is j,,st of the form (exp(G/H))/G. So, if Con- 
jecture 1.8 were proved, the Banach-Mazur compacta would be just of the form 
(exp(G/H))/(; (with (1 = O(n) and H = O(n- 1)). On the other hand, the 
st)ace (ext)Ç)/G il, "Vest's pro|)lem is also of the fo,',,, (exp(G/H))/G (with H 
the trivial subgroup). This shows how close are, il, fact, Pelczyfiski's probleln and 
Abst "s problcm. 
|h're is (mr fourth result on Bmm('h-Mazur compacta, which is proved in Sec- 
tion 7: 
Theorem 1.9. exp0 S'- and La(n) bave the saine O(O-homotopy type. 
It follows ilmnc(liately ff'oin Thcoreln 1.9 that Lo(n)/H and (exp0 Sn-)/H 
bave the saine homotopy type for any closc(l subgroup H C O(n). This result and 
Theorem 1.3 constitute esscntial steps in proving Conjecture 1.8. However, we do 
no, prove Cnjecture 1.8 in this paper. The only step we lack to colnplete its proof 
is that (exp0 S'-I)/H. n _> 2, is a Q-malfifold. The de, ails of this reduction are 
also presented in Section 6 (Theorem 7.9). 
The paper is divided as follows: 

§1. Introduction. 
§2. Prelinlinaries. 
§3. Proof of Theorem 1.1 and its corollaries. 
§4. The G-nerve. 
§5. Proofs of Theorems 1.3 and 1.4. 
§6. Proof of Thcorem 1.6. 
§7. Proof of Theorem 1.9 and reduction of Conjecture 1.8. 

2. PRELIMINARIES 

For a given topological group G, we denote by G-A(N)IR (resp., by G-A(N)E) 
the class of all G-equivariant absolute (neighborhood) retra«ts (resp., extensors) for 
ail metrizable G-spaees. These concepts are straightforward extensions to the case 
of G-spaces of the corresponding concepts of ordinary A(N)R's and A(N)E's (see, 
for example, [21-[61). We refer to the monographs [11] and [26] for basic notions of 
the theorv of G-spaces. 
If G is a topological group and X is a G-spaee, for any x  X we denote the 
stabilizer (or stationary subgroup) of x bv G, = {g e G I gx = x}. 
For each subgroup H C G, the H-fixed point set X[H] is defined to be the set 
{xeX I HcG.}. 
The familv of all subgroups of G that are conjugate to H is denoted by (H), i.e., 
(H) = {gHg-l g e G}. We will call (H) a G-orbit type (or simply an orbit type). 



VEST'S PROBLEM AND BANACH-MAZUR COMPACTA 3383 

For two orbit types (HI) and (H2), one says that (H1)  (H2) iff H1 C g-lH2g for 
sonle g E (7. The relation -< is a partial ordering on the set of all orbit tvpes. Since 
For a subset S C X, H(S) denotes the H-saturation of S. i.e., H(S)= {hs I h  
H, s e S}. In particular, H(z) (lenotes the H-orbit {hz e X ] h e H} of z. The 
H-orbit space is denoted 1)y /H. In particular, X/G denotes the orbit space of 
X. 
By G/H we will denote the G-space of cosets {gH I 9 e G} mder the action of 
G induced by left translations. 
A contiluous lna I) f  X  1" of G-spaces is said to be equivariant or G- 
equivariant or, for short, a G-map, if f(9 x) = 9f(x) for all 9 G G,  G X. An 
equivariant lnap f  X  I is said to be isovariant (or G-isovariant) if G=G() 
for all x G X. 
A compatible metric p on a G-space is called invariant or G-invariant if p(gx, 9) 
=p(x,y) forallx, yG X andgGG. 
If X is metrized 1,y a G-invariant metric p, then the formula ç(G(x), G(y)) = 
inf{p(x', y')[ x' e G(a'), y' e G(y)} defines a metric ç, compatible with the quotient 
topology of X/G, whenever G is a compact group. 
Let us recall the well-known and ilnl»ortant defilfition of a slice [26]: 
Definition 2.1. Let G be a topological group, H C G a closed subgroup and X a 
G-space. A subset S C X is called an H-slice in X if 
(1) S is H-invariant, i.e., H(S) = S, 
(2) the saturation G(S) is open in X, 
(3) ifgGH, thengSS=0, and 
(4) S is closed in G(S). 
If in addition G(S) = X. then we say that S is a global H-slice of X. 

The following is one of the ftllldalllelltal results in topological transfornmtion 
group theory (see [26, Corollar.v 1.7.19, Corollary 1.7.20 and Theorenl 1.7.7] or 
Il l, Ch. II. §§4 and 51): 
Theorem 2.2 (Slice theorem). Let G be a compact Lie group. X a Tychonoff 
G-space and x  X any point. Then: 
(1) There exists a G-slice S C X such that x  S. 
(2) (Gu)  (G) for each point y e G(S). 
(3) Th« «xists a unique G-map f: G(S)  G/G, such that S = f-(eG,). 
In [9], using the classical result of John [21] on the minimal-volume ellipsoid, it 
was proved that L(n) is a global O(n)-slice for the GL()-space B(n). In colnbilm- 
tion with a result of H. Abels [1, Theoreln 2.1] this yields the following theorem, 
which we will need in the sequel: 
Theorem 2.3. There is an O(n)-equivariant traction r : B(n)  L(n) such that 
r(A) belongs to the GL(n)-orbit GL()(A) for every A  B(n). 
In [9, Corollary 2] it w proved that J(n) is a compact O(,)-AR, and since L(n) 
is O(n)-homeomorphic to J(n) [9, Remark 1], we bave the following result that will 
offen be used in what follows: 

Theorem 2.4 ([91). L(n) is a compact O(,)-AR. 
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Let fo, f : X -- X' ]»e G-mai»s. A G-homotopy of f0 into f is a homotopy in 
the ordinary seuse which is a G-map ai each stage of the deformation. A G-space 
X is callcd G-contractibl(' if there is a G-fixed l)oint  G X such that the constant 
G-ma I) X  {.} and the identity ma I) 1 - are G-honlotopic. A G-lnap f : X  } 
is a G-homotopy equivalence if there is a G-map f' : }"  X such that f'f is 
G-homotopic to lx and fff is G-homotopic to 1 . 

Theorem 2.5 ([20]). Let G be a compact Lie group and f : T --, Z a G-map 
of G-ANR's. Then f is a G-homotop:eq cquivalcnce iff for each closed subgroup 
K C G. the restriction of f fo thc K-fi.rcd point set T[Iç] is an ordinary homotopy 
equivalence. 

Remark 2.6. In [211, Proposition 4.1] the result originally was stated for paracom- 
pact G-ANE's (even in its filwrwise form), tlowever, the proof in [20] serves for 
metrizable G-ANI/'s as well. 

Yct another basic result for this paper is the following. 

Theorem 2.7 ([6], [7]). Let G be a compact .qro,p. N C G a closed normal sub9roup 
and X a G-ANR (rcsp.. a G-AR). Tben the N-orbit space X/N. endowed with the 
induc«d action of the quotient 9roup G/N, is a G/N-ANR (resp.. a G/N-AR). In 
particular, X/G is an ANR (resp., an AR). 

t/ecall that for a inertie space (X. d), the Hausdorff metric dH on exp X is defined 
by the formula 

dH(C,D) = max { supdist(x,C), supdist(y,D)} 
D yC 

for C.D G expX. 

The topology generated by dH is an invariant of the topology of X (it does not 
depend on d). 
If G is a compact group raid X is a metrizable G-space, then the fornmla 

(9, A)'--'gA: 9.4={ga[aA}, for allgG. AexpX 

defines a COlltiiltlOilS (7-action on exp X; so exp X naturally becomes a (7-space In 
this case the conplement exp0 X = (exp X) \ {X} is an open invariant subset of 
exp X. Clearly, if d is a G-invariant metric on X, then dH is a G-invariant metric 
on exp X. 
For the boundarv of a set A C X we will use the notation OA. 
Throughout the paper we will use the following standard notation: 

' z <_ 1}, the Euclidean unit ball: 
B  = {(x ..... :r)   I xï +... + x 

2 = 1}, the Euclidean unit sphere: 
s - = {(z,...,x) e  I +"+ z 

A' = {(t0 ..... t,)  IR '+ [ t > 0, t0 +-.-+ t, = 1}, the standard closed simplex: 

Q = H {I,. I Ik = [0.11}. 

the Hilbert cube. 
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3. PROOF OF THEOREM 1.1 AND ITS COROLLARIES 
We first prove thc following equivariant version of Curtis' generalization [13, 
Theorem 1.6] of the well-known Wojdyslawski Theorem [35], which is just the im- 
plication (1)  (2) in Theorem 1.1. 
Rccall that a metric space X is continuum-connected if each pair of points in X 
is contained in a subcontinuum. X is locally cotinuum-connected if it has an open 
ba of contilmum-connected subsets [13]. 
Proposition 3.1. Let G be a compact group and X a locally continuum-connected 
(sp., connected and locally continuum-conected) metrizable G-space. Then exp X 
is a G-ANR (resp., G-AR). 
Proof.  shall consider only the "G-ANR'" ce. The "G-AR'" case is similar. 
If exp X is a G-ANR, theu it is also an ANR, and then by Curtis' theorem [13], 
X is locally continuum-connected. 
Now assume that () p) is a metric G-space with p an invariant metric on Y, A 
a closed invariant subsct of Y, and ç : A  exp N a G-map. By Çurtis' theorem 
[13], exp X is an ANR. So f has a contimmus extension ç : U  exp X defined on a 
neighborhood U of A in Y. By compactness of G, therc is an invariant neighborhood 
V of A contained in U. Set 
(u) =  -ç(9) fo v¢y  z ç 
9G 
We claire that the map F : V  exp X is a well-defined continuous G-equivariant 
extension of f. Indeed, for every y  V the set {g-lf(gy) I g  G} is a compact 
subset of exp X because it is the image of the continuous map  : G  exp X, 
a(g) = g-lf(gy). Therefore, the union 9¢ g-lç(gy) is a compact subset of X, 
i.e., F(y)  exp N. 
Further, if t, g  G and h = gt, then 
gG hG hG 
showing the equivariance of F. If a  A, then ç(ga) = f(ga) for all g  G, and by 
the equivariance of f we will then have 
showing that F extends f. 
Let d be an invariant znetric on X. 
To see the continuity of F, we fix y0 E V and e > 0 arbitrary. By continuity of 
ç, for each g E G there is a g > 0 such that 
By compactness of the orbit G(y0), its open cover {O(gyo,3g)] g e G}, where 
O(z, r) stands for the open r-ball in X centered at z, admits a finite subcover 
{ O(glyo, g ), .... O(gkyo, 3g, ) }. 
Let 3 = min{3g,,...,3g}. We are going to check t5at then d,(E(y),F(yo)) < e 
whenever y E V and p(y, Yo) < . 
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h(h,ed, fol every g (5 G, gy0 belongs to ail element of the cover 
Without loss of generality, one ('an assume that 9Yo  0(91Yo, m )" Then for every 
  0(90, ) we have 
P(gu. glYo) g P(gi.g.o) + P(ggo, g,Yo) = P(Y, Yo) + P(gYo, glYo) g 5 + Sa, g 25 m. 
Thus, gYo,gY  O(giYo,25a), which implics that dH(ç(g,),ç(gyo)) < e. Bv the 
invariancc of d, this yields 
d H ( g-  ç(gy), g- ' ç(gYo) ) 
whi«h in tnrn implies that 
gG 
Thus, dH(F(y). F(yo))  e for ail y C ()(y,). 5). l)roving the «ontimfity of F at 
thc point y0- 

R« mark 3.2. R('call that a hypersl)ace c e C exp X that satisfies the following condi- 
tion is callcd an inclusiou h!lp«rspace: if/3 (5 exp X and A (5 ce is su«h that A C B, 
thon B (5 ce. As is clora" from thc I)r()of, Proposition 3.1 renmins true also for any 
invariant inclusion h.vperspace £ instcad of exp X. 
(2) == (3) follows directly from Theorcm 2.7. 
Since everv ANR (resp., Al:l) is locally continuum-commcted (resp., commcted 
and locallv continuum-commcted), the implication (3) == (1) follows from the 
following result. 
Proposition 3.3. Let G be a compact 9roup and X a metric G-space. Then: 
(1) X is connected iff (expX)/G is comected; and 
(2) if in addition G is a Lie group, then X is locally continuum-connected iff 
(exp X)/G is locally path-connected. 
For the proof we need the following thr(,e lemmas. 
Lennna 3.4. Let G be a compact group, and A a G-space containin 9 a comected 
invariant subset (e.g., G connected or X[G] ¢ 0). Then X is connected iff X/G is 
connected. 

Pro@ Only the "if" part requires a proof. Let C C X be an invariant connected 
set. Assume the contrary, that X/G is connected and X is not connected. Then 
X = A1 U A2 where the Ai, i = 1, 2, are nonempty, disjoint, closed-open subsets of 
X. Since C is connected, one (and only one) of the sets Ai, i = 1,2, contains C. 
Suppose C c AI. By compactness of G, the orbit map p : X -- X/G is open and 
close& Hence the set p(A) is a nonempty open-closed subset of X/G. Besides, by 
the invariance of C the set p(A2) is disjoint from p(C); so p(A2) ¢ X/G, which 
contradicts the connectedness of X/G. [] 

Lemma 3.5. Let G be a compact Lie group, and X a G-space containing a path- 
connected invariant subset (e.g., G connected or X[G] :¢ 0). Then X is path- 
connected iff X/G is path-connected. 
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Pro@ Only the "if" part requires a proof. Let C C X be an invariant path- 
connected set. It suffices to show that each point x E X can be joined with a point 
of C. Let «E('andlc l: [0.1]--X/Gbeapathwithl(0)=p(x), /(1)=p(c), 
where p : X -- X/G is the orbit projection. By [11, Ch. [I, Theorem 6.2] there is 
a lifting l' : [0, 1] -- X, pi' = l. Since l'(O) belongs to the orbit of x, there is g  G 
su«h that x = gl'(O). Then thÇ path gl' : [0, 1] -- X, (gl')(t) = gl'(t), «onnects x 
and g/'(1). But gl'(1)  C, becausc ff(l}  C and C is invariant. This completes 
the proof. [] 

Lemlna 3.6. Let G be a conpact Lie group. Then a G-space X is locally path- 
connected iff X/G is locally path-connected. 

Pro@ Only the "if" part requires a proof. Let x  , and let U be a neighborhood 
of x. Since the action of G is continuous and G is locally path-connected, one tan 
choose a path-connected neighborhood O of the mfity in G and a neighborhood I" 
of x such that OI" = {gv I g E O, v  I "} C U. \Vithout loss of gcnerality, one tan 
assume that I  is G:-invariant. Since G is a compact Lie group, there is a Gx-slice 
R containing the point x [26. Corollary 1.7.17]. Then the set T = R VI V is also 
a G-slice containing x. Since X/G is locally path-connected, the neighborhood 
p(G(T)) of the point p(x) contains a path-connected neighborhood t" of p(x) in 
X/G. Let Y = p- ('). Then " is a G-invariant neighborhood of the orbit G(x) 
in X. Clearly, the set S = T  t contains x and is a global G-slice for the G- 
space Y. Therefore the orbit spaces S/G and }'/G = Y are homeomorphic [26, 
Proposition 1.7.6]; in particular, S/G is path-connected. By Lemma 3.5, now S is 
path-connected as wcll. Consequently, OS is path-connected. Since OS is an open 
neighborhood of x and OS C OT C 0I" C U, we are done. [] 

Proof of Proposition 3.3. 1. It is well known that X is connected iff expX is so: 
this is proved, for instance, in [23. Proposition 5.3.10] for X compact, but the 
saine proof is valid for noncompact X as well. Hence, it remains to show that the 
connectedness of (exp X)/G implies that of exp X. If (exp X)/G is connected, then 
the invariant subset F of exp X consisting of all invariant, compact subsets of X is 
connecte& Indeed, the continuous surjection exp X -- exp(X/G) induced by the 
orbit map X -- X/G is invariant, and hence, it induces a continuous surjection 
(expX)/G -- exp(X/G). Therefore, if (expX)/G is connecte& then exp(X/G) 
is so, and since F is homeomorphic to exp(X/G), it is connected too. Thus, F 
is an invariant, connected subset of the G-space exp X, and the connectedness of 
(exp X)/G implies, by Lemma 3.4, the connectedness of exp X. 
2. Respectively, X is locally continuum-commcted iff expX is locally path- 
connected (see [13, Lemna 1.4] and the final part of the proof of [13, Theorem 
1.6]). By Lemma 3.6, exp X is locally path-connected iff (exp X)/G is locally path- 
connected, and the proof is complete. Thus, Theorem 1.1 is completely proved. 
Corollary 3.7. Let G be a compact, metrizable, locally path-connected grop, and 
H c G a dosed sbrop. The. (xp(G/H))/G is a** nNl. If in addition G/H is 
connecte& then (exp(G/H))/G is an AR. 
Pro@ Under the hypothesis the coset space G/H is locally path-connected, and 
hence, locally continuum-connected. Besides, since the quotient map (7 -- G/H is 
perfect, G/H is metrizable too (sec, e.g., [15, Section XI, Theorem 5.2(3)]). Then 
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I)y Proposition 3.3, exp(G/tl) E G-ANR; if in addition G/Il is connected, then 
exp(G/H)  G-AR. Now thc rcsult follows imnmdiately from Theorem 2.7. [] 

Sinee every compact Lie group (7 is locally path-colmected (moreover, it is a G- 
ANII [26, Corollary 1.6.7]), we get from Corollary 3.7 the following positive answer 
to the first question of West's Problem: 

Corollary 3.8. Let G be a compact Lze group. Then (exp G)/G is an ANR. If in 
addition G is cotmected, then (expG)/G is au A R. 

In COlnbilmtiol with [17, Corollary 2], Theorem 1.1 yields also the following fact. 

Coroilary 3.9. Let G bc a finite group acting on a nondegenerate Peano continuum 
X. Then the orbit space (expX)/G is a Hilbcrt cube. 

4. THE (7-NERVE 

In this section we dcvolo I) a necessary techlfique involving the notion of a G- 
lmrve. Thc rcsults proved |u.re wil| be applied in the next section. 
Folh)wing Matuln«)to [221, we (lefiue the G-ncrve JV'(/) of a G-normal cover/'. 
Let G ])e a COlnpact Lic group and  = {Hul p G M} a fanfily of closed 
subgroups of G. The Mihlor join J of the falnilv of «osets {G/Hu] #  M} is 
defined as follows. Let J bc the following set: 

{(t,,9vH,),¢^, [-[ (IxG/H,) lt,,¢O foronlyfinitep's and 

We let 

(t. g.t.).^« ~ (s.. 

ifft u = s u for ail p G.A1, and gu u= ht, H  whenever t u ¢ 0. Then ,- isan 
equivalence relation on 3", and we shall denote by 3" the quotient set 3"/,-. 
In what follows we shall use the convention ' tu9uH u for the equivalence elass 
of the point (t u, guHu)ue^t. The nUlnbers t, are called baryeentrie coordinates of 
the point  tu9,H »  3". 
For any finite subset {P0 .... , p, } C M, we consider the following subset of 3": 

Observe that each G/Huo  ...  G/Hu, with its quotient topology is a compact 
met.rizable space, which is called the Milnor join of the spaces G/Huo, .... G/Hu, 
(see [21). 
We t.opologize 3" by the weak topology with respect t.o the fanfily of all finite 
subjoins, i.e., a set U C 3" is open in 3" whenever U C (G/H,o  ..., G/H,,) is 
open in G/H,o ,..., G/H,, for any filfite subjoin G/H,o ,.-.  G/H,,, C 3". It 
lS easy to check that 3" beeomes a G-space if we define the action of G as follows: 

g( Z tuguH') = Z tugguHu" g G G. 
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Next, if .quo Huo  G/H,o,. ., gu. Hu,  G/Hu,, are fixed elements, then we will 
denote by (go Ho, .... g,, H } the subspace 
 tpgpHp e  ] tp = (} for l' 
, 
a.nd 9, H = 
of G/H.o * "" * G/H., which is called an -cell. 
Let G be a conlpact Lie group and X a G-space. For each index 
be a closed subgroup of G. and let S. be an H.-slice in X. Then the falnily 
H = {(.qSp,Hp)[ g e G, p e Al} 
consisting of tubular sli«e-sets gSp with companion groups Hp is called a G-normal 
cover of X if the family of opcu tubes {G(Sp)[ p  AI} covers X and there exists a 
locally finite invariant partition of mfity {çp  X  [0, 1][ p  AI} subordiuated to 
H, i.e., each p is an invariant fuuction with çç((0, 1]) C G(Sp) aud the supports 
{1((0, 1])[ p  Al} constitute a locally finite fmnilv. 
Let (14) be the ordinary nerve of thc invariant cover {G(Sp) [ p  M}. In the 
scquel we will denote by (P0 ..... p.) the simplex of ,(H} coustituted by the sets 
G(Spo ) .... ,G(Sp). Let fp " G(Sp  G/Hpbe the corresponding G-map with 
f(eH,) = S u (see Sli«e Theorem 2.2). For any simplex L={p0 ..... p)  (H), 
we defiue the following sui)set of the product 
i:0 
i=0 
It follows from the equivariance of fro that F is an invariant subset of the 
G-space  
i=0 
Denote by  the fmnily of ail these sets FL. Let (L, FL) be the subset of the 
finite subjoin G/Hpo * ... * G/Hp of  consisting of ail those points  tgH,, 
i=0 
for which (goH, o,...,gnHp,) e & aud (t0 ..... t,,) e A'. Clearly, A(L,&) is 
invariant in G/Hpo  ... * G/Hp,, and hence, in 
XX call A{L, Fe) a G-,-simplex over the simplex L along the set FL. The 
homogeneous G-spaces G/Hp ..... G/Hp, ae called G-vertices of A(L, FL). 
The G-nerve of the cover U is. by definition, the ullion 
equipped with the topology induced ffoto . It is not difficult to check that 
the topology of (U} is the weak one with respect to its closed, invariant cover 
{A(k,&) I L e (U),& e *}. That is to say, a subset II" C (U) is open iff 
II"  A(L, &) is open iu A(L, &) for each G-simplex A(L, FL) in (H). Since 
() is ai1 invariant subset of , it becomes a G-space with respect to the action 
induced from . 
The G--skeleton (U)' of (U) is defined to be the union of ail G-k-simplexes 
in (U) with k 5 
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If .qHx is a vertex of t|le G-nerve ,)V'(H), t|len its star St(gHx,,N'(Lt ) is defined 
to be the lmion of all cclls fir which gH is a vertex. The G-carrzer of a point 
z  (14) is defined to bc the smallest G-simplex of () containing . The cell 
in the G-carrier that conta.ins a" is ealled the caier of x. 
Recall that a cover  of a stmce X is callcd a star-refinement of a cover V 
whcnew, r for eVelV U   there exists an elelnent I" G V that contains the star 
St(U,14) of U with respect of/g: here St(U.14) = {II 14 ] II' U ¢ }. 
Lelnlna 4.1. Lct X bc a paracompo«t G-space. Then for each open cover V of X 
there eists a G-normal cover 14 = {(9S, H)  9 G G, A  A} of X such that  is 
a star-refinement qf F. 
Pmof. Since X is Imra«ompa«t, one can çhoose open «overs lfi and H2 of ç such 
that lgi is a star-refincmcnt of 112 and lt2 is a star-refinelnent of ç. 
Lel us d,note l»v U the subset of X x X consisting of ail those pairs (x, ) such 
that th«re exists au clenlcnt ()  U that contains both .r aih 9. Clearlx U is an 
open neighborhood of the diagonal  C X x N. Bv compactness of G. there is an 
invariant neighborhood !" of  in X x X su«h that !" C U. Define 
={V[]]..¥}. wh«,,'«  [.]={:XI(, )'}. 
Then  is an open G-cover of X, and V[m] C U[m] = St(wl/) for each   X. 
Since St(z,l) is contained in an element of e, we iufer that W is a refinement of 
2, and hen«e, a star-rcfinement of . 
Next, we fix on each orbit G(w C X a point, say   G(«). and choose an 
elemcnt I   su«h that .r  llç. By the continuity of the action of G on X 
there exist a neighborhood O of the unity in G and a G-invariant neighborhood 
N of « in X such that O«N C II. BS' Slice Theorem 2.2, there exists a G- 
slice Q, such that m  Q,. Then the set S, = Q  «XÇ is also a G-slice, and 
«  S C XÇ. Hence O«S C llç. % dcfine  to be the totality of all these sliee- 
sers (9S, G,), g 6 G. G(«) 6 X/G. Since the orbit map X  X/G is closed, we 
sec that X/G is paracompact too [15, Section VIII, Theorem 2.4]. This implies that 
the invariant cover {G(S)} admits a locally finite partition of unitv subordinated 
to . and hence.  is an open G-normal cover. Since 9S C 9Ilç and 9Ilç  W, 
we conclude that  is a refinement of W, and since W is a star-refinement of ç, we 
infer that  is a star-refinement of ç. 

Lemma 4.2. Let Y be a G-space, and let lg = {(gSu,Ht, ) ] g  G.p E M} be a 
G-normal cover of  ". Then for each locally finite invariant partition of unity subor- 
dinated to lg, there ezists a G-map p: }" -- N'(lg) such that p-1 (St(gHt,,,%r(lg))) C 
G(S) for any g  G and p  M, where 

AE/I 

Pro@ Let {çu : Y -- [0, 1] ] #  M} be a locally finite invariant partition of unity 
subordinated t.o L/, i.e., çl((0.1]) C G(S.) for all /  M. Then we define the 
canonical G-lnap p : Y  N'(/) as follows. 
Since {ç. : Y --, [0.1] ] I' e BI} is locally finite, for each 9 e t" there are only 
a finite number of indices, say #0, -.-, Itn . such that çm (9) ¢ 0, i = 0 .... , n. Let 
(f.o(9) ..... f. ()) be the corresponding n-cell in G/H.o ....  G/H.,,. Then by 
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definition, p(y) is the point of (fuo(Y) .... , fu, (Y)) with the barycentric coordinates 
ç.o(Y),-" ", ç.. (Y), i.e., 
 claire that p is continuous. For, let Yo G }" be an arbitrary point. Using the 
local finiteness of he partition of uniy {çu ] P G M}, we take a neighborhood V 
of Yo in Y such that only for a finie number of indices p.o .... , P'm is çu,(Y) ¢ 0 
for y G V. Then 
Now the contimlity of p in V follows from the continuity of the maps fp, and 
i=0 ..... m, in I'. 
For a space X we will denote by (X) the subsct of expX that consists of all 
those sers A c X that have at most n elements. By (A) we shall denote the 
union U    (X). 
Lemma 4.3. Let P be a polyhedron and P it.s 1-dimensional skeleton. Then there 
is a continuous map   P  ff(pl) such that 
() ç(z) = {z} b  z e P, 
(2) if r is the carrier of x  P and dira r = n. then ç(x) is contained in the 
1-skeleton of r and contains at most 3 -1 points. 
Proof. Let P, n  1, dcnote the n-skeleton of P, and let   pi  p be the 
identity lnap. From the proof of [23. Proposition 8.4.2], we get the following fact. 
Claire. For every n OE 1, there is a contimous map ç  P  (P) such that 
a. ifr isthecarrierofx Panddimr= n, then(x) iscontainedinthe 
1-skeleton of r and contains at most 3 -1 points, and 
b. +1 extends ç for all n OE 1. 
Now we define the required map ç to be equal to  on the n-dimensional skeleton 
p. 
In the next lemnm, for a given simplex L = (o ..... } C (U), a given n-cell 
{gpoHpo,..., gpH,} C (U) and the corresponding G-n-simplex a = (L, FL), 
we shall use the following notation: 
where we uoe the saine notation 0  for the ordinary boundary of the standard 
simplex A . Correspondingly, 
o« = { .,. I ( ..... t) e oa , (..,...,..1 e F . 
i=0 
In what follows we shall need the following equivariant version of Lenmla 4.3: 
Lemma 4.4. Let (U) be a G-nee and F its G-l-dimensional skeleton. Then 
the is a G-map R "(U)  YOe(F) such that 
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(2) if s is thc carrier of .r  Af(14), thon R(x) is contained in thc 1-skcleton of 
s. More pr«cis«Ig, if dira s = n. thon R(x)  s- (s). 
Pro@  are goi,lg te apply Lcmma 4.3 at»o, hl out case P is the polyhedron 
acconlpanying thc G-ncrve (/4), i.e., the polyhedron (). Let K be its 1- 
skelet(m raid ç  P  (N) thc contiuuous nmp rioto Lemnla 4.3. 
Let x  ,ç(l) and let the G-simplex 
a= 5(L, Fr) C G/H,o ,...,G/H,, 
n 
t,e the G-carrier ,f x. Th¢'n .r =  tigmHm, where s = {goHo ..... g.H.) is 
the carrier of x. 
Define the map R : (U)  U(F) by etting 
R(w) 
i=0 
Sin«e (to ..... t,) belongs te s--'(P). we sec that R(x) belongs te Us.-,(s) C 
Us,-(F). Contimfity and equivariauce of R are evideut frein the definition of R. 
PIoperties (1) and (2) h»llow fr»m the analogous properties in Lenmm 4.3. 

5. PIOOFS OF THEIEMS 1.3 AND 1.4 
We shall give a sequence of lenmlaS cuhninating in proofs of Theorems 1.3 and 
1.4. 
In this section d will alwavs dcnote thc Euclidean metric on I 
Bv (l) we will denote the subset of LOt ) consisting of ail compact convex 
bodies A such that the contact cet 0A F)OB n has an empty interior in the boundary 
sphere S'-  = OB '. 
Lemma 5.1. For each ¢ > 0 and each body E  Le(n) there exists a body D  7)(n) 
such that dH(E.D) < e and the O(n)-stabilizçr O(n)oe of E coincides with the 
O(n)-stabilizer O(I)D of D. 
Pro@ Let r : B(n) -- L(n) be the O(n)-equivariant retraction frein Theorem 2.3. 
Because of compactness of L(n) (Theorem 2.4), one can find a real 0 <  < s/2 
such that dH (r(A), A) < e/2 for all A belonging te the -neighborhood of L(n) 
in/30t ), where dH denotcs the Hausdorff metric on 
Let K be the stabilizer O(n)E. If follows frein Slice Theorem 2.2 that there is a 
rem 0 < q <  such that the inequality dH(E, X) < q for X  L0(n) implies that 
the stabilizer O(n)x is conjugate te a subgroup of K, i.e., (O(n)x) _ (K). 
Choose a centrally symmetric, convex polyhedron P C ll n with a nonempty 
interior, such that dH(E, P) < q, P C E and ail the vertices p ..... Pk of P lie on 
the boundary DE. Then the convex hull 
/I I = conv (K (PI) ['-J " " " ['-J IÇ (Pic) ) 
is a cent.rallv symmetric, compact, convex, K-invariant subset of l n. Since it 
contains P, we sec that I has a nonempty interior in ll n, and se I 
,Ve clailn that the boundary OIIl does net contain an (n- 1)-dimensional elliptic 
demain, i.e., an open subset V c 0I which is at the smne rime an open subset 
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of some (- - 1)-dimensional ellipsoid surface lying in IR '. It suffices to show that 
none of the orbits K(pi), i = 1,...,k, contains an (n - 1)-dimensional elliptic 
domain. Assume the contrary, that some K(pi) contains an (n -- 1)-dimensional 
elliptic domain V. Since h'(pi) lies on the (n- 1)-dimensional sphere 0B(0, 
centered at the origin and having the radius Ilpgll, then V should be, in fart, a 
domain of the sphere OB(O, IIPII)- Since h(p) is homogeneous and compact, we 
conclude that there are finitely many open subsets Va ..... Vu of K(pi) such that 
K(pi) = V1 U-.. U Vu, where each I is homeomorphic to 1". Next, by the Domain 
Invariance Theorem (see, e.g., [27, Ch. 4, Section 7, Theorem 16]), earh Ij should 
be open in the sphere OBO, IIP, II), and hence, the union 1 U -.- U çÇ is open in 
OB(O, IIPll)- But V1 I..)---U V n is also closed in 0B(0, IIPII), because it is equal 
to h'(pi). Now, by connectedness of OB(O, IIPilI), it then follows that Iç(pi) nmst 
be the whole sphere OB(O, IIPII)- Consequently, Iç arts transitively on the sphere 
OB(O, IIPII), and hence on the unit sphere S'-1. This contradiction proves the 
claim. 
hl particular, the contact set of AI, which is hy definition the intersection of 
with the boundary of the L/Swner ellipsoid i(M), also does hot contain an elliptic 
domain. 
Now consider the body D = r(M)  L(). Since D = T(M) for some linear 
nondegenerate operator T : IR ' - IR n, the contact set OD  OB  is just the image 
under T of the contart set OM ç) O(i(M)). Consequently, OD f-) OB ' has an empty 
interior in the sphere S'- = OB ', and hence, D e P(n). Sinee dH(P, E) < q and 
P C M C E, we see that dH(M, E) < 71 < 6. Consequently, dH(D, E) < e, which 
implies in turn that (O(n)D) _ (Iç). Since M is K-invariant, we have K C 
and since r is O(n)-equivariant, O(n)^t C O(),-(^t) = O()D. Thus K C O(n)D, 
which implies in combination with (O(n)D) __ (h') that O(ï)D - K. [] 
The following lemma is the key in the proof of Theorems 1.a and 1.4: 
Lemma 5.2 (Key lemma). Let e > O, and let 12 be the e-cover of Lo(). Then 
there exist a (3-normal cover/d = {(gS» H),) I g  (3,   A} of Lo(n) and G-maps 
p: Lo(n) - Af(U),  : Af(U) - P(n) such that 
(1) for each gSx  bt there is an element I'(g, )  IL/2 with gSx C V(g, 
and ¢b(St(gH,Af(bl))) C V(g,). and 
(2) the composition ¢bp is e-close to the identity map of Lo(n). 
Pro@ By Lemnm 4.1, there is a (3-norlnal cover 
b/= {(gS, H)I g  GA  A} 
of L0(n) that is a star-refinement of Fe/4. Fix an invariant locally finite parti- 
tion of unity {çx}xe^ subordinated to b/. Let p : L0(n) --, Af(b/) be the G-map 
corresponding to {çx}xe^ (see Lemma 4.2). 
For every 9S  bt we choose an element V(9, ,)  12e/ such that 
(5.1) St(gS»bt) C V(g,A)/2, 
where V(g, )/2  Fe/4 denotes the open ball in Lo() concentric with V(g, ) and 
having hall the radius of V(g, ). 
Now we define the map q) :Af(b/)  T)(n) as follows. 
First we define a G-nap q : F  7)(), where F is the G-l-skeleton of Af(b/). 
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For each G-vertex G/HA E Aç(ld) with ttA the subgroup corresponding to the 
HA-slice SA E L/, we select a point Xx  Sx such that the stabilizer Gx coincides 
with the group H. 
By Lemma 5.1 above, we choose a body A  P(), s/4-close to X and having 
the stabilizer Gax = H. 
Define the G-map q : G/H  G(A) C P(n) by setting 
q(gH) = gA for every 9H  G/H. 
The inclusion H C GAx guarantees that q is a well-defined G-map. X% aim to 
extend q equivariantly to the G-l-skeleton F of oEr(). It suffices to extend q over 
cach G-l-simplex of F. Let  be a G-l-simplex of F with thc G-vertices G/H and 
G/H,.  first define two G-maps s :   B(n) a.nd s :   B(n). Indeed, 
let y = tgoHx + (1 - t)gtH, bc an albitrary point of , where 90, g  G and 
% set 
s(y) = tgoAx + (1 - t)g.4, and s() = goAx. 
Recall that hcrc fil'+ (1 -t)Z means thc hlinkowski convex combination of the 
convex scts II', Z C Nn, i.e., 
t +(-t)z= {t+(-t) ]e  ze z}. 
Clearly s and s are continuous G-maps from A to B(n). 
Dcfine q'(y) to bc the convex hull of the uni(m s (y) U s(y). 
Since s(y)U s2(y) depends contimously upon y  , the continuity of q now 
follows from the continuity of the convex hull operator (sec [31. Theorem 2.7.4(iv)]). 
Analogously, let q'(y) be the convex hull of the union 
gA U (tgoAx + (1 - t)gA). 
For the saine reon, q'(y) depends continuously upon y 
Now we pte q' and q" to dcfine the desired map q: 
{ q((1 - 2t)goHx + 2tgH), if 0 < t < 1/2. 
q(t«x+(-«)'») = q"((e«-- 
Since (1 - 2t)goHx + 2tg H and (2t - 1)gl H + (2 - 2t)goHx depend contin- 
uously upon tgoHx + (1 - t)glH   and q'(glA) = q"(goAx), we sec that the 
continuity of q' and q" implies the continuity of q. The equivariance of q is evident. 
Let us check that q(y)  P(n). 
Indeed, since for each t  [0, 1/2], 
A c q(tH + ( - )H.) c B , 
we infer that q(tgoHx +(1-t)g H) belongs to L0(n). Further, for each t  [0, 1/2], 
the contact set of q(tgoHx + (1 - t)gHp) is a subset of the contact set of g0Ax, 
and hence, it h an empty interior in the sphere S -. 
Analogously, for t  [1/2, 1], 
1 Ç q(tgoHx 
which implies that q(tgoHx + (1 - t)gH) belongs to L0('). The contact set of 
q(tgoHx + (1 - t)glH) is a subset of the contact set of gA, and hence, it also 
has an empty interior in the sphere S 
Thus, we have provcd that for arbitrary y 
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By Lemma 4.4, there is a G-map R " AF(L/) - 9(F) satisfying conditions (1) 
and (2) of Lemma 4.4. Observe that fol" every y  AF(L/), R(y) is a finite collection 
of elenlents of F. Now we defille ' -AF(L/) -- exp n by 
 '(y) = [_Jq(n(y)) = [_J q(),   AF(LC. 
Then 1 is well-defined, and by [23, Corollary 5.3.7] it is contilmous. The equi- 
variance of ' follows from that of R and of q. 
Let. (y) be the convex hull of the set '(y). Since '(y) C (y) C B', we infer 
that (y)  Lo(n). 
The continuity of  now follows from the continuity of 1 and of the convex hull 
operator (sec [31, Theorem 2.7.4(iv)]). 
Next, since '(y) is a finite union of bodies from P(n), its contact set, i.e., the 
intersection ÔB  f 0(y), is the finite union of the contact sets of these bodies 
from P(n). Therefore, 0B  ç 0êP'(n) has an empty interior in 0B . It remains to 
observe that (y) and '(y) bave the saine contact set, and hence, (y)  P(n). 
Let us check that p(St(gH,AF(L/))) C l(g, A) for all g  G, A  A. Here we 
need the following: 
Claire. Let O(X, a), a > 0, be the open a-ball in /(n) centered at X  /(n). 
If y = tgoH + (1 - t)gH»is a point of the 1-cell (goH,gH» such that the 
corresponding sers goA and gl A»belong to O(X, a), then q(y)  O( X, a) ç Lo(n). 
Proof of the Claire. First of all we observe that O(X, a) is 'aJways a convex set in 
B(n), i.e., if Y, Z  O(X, a), then fol" every t  [0, 1] the convex body tX + (1 - t)Z 
belongs to O( X, a). 
Hence, st(y) = tgoA + (1 - t)A u  O(X, a), due to the convexity of O(X, a). 
One also has s2(y) = goA  O(X, a). 
On the other hand, according to ([31, Theorem 2.7.4(iv)]), the convex hull op- 
erator is non-expansive. So fol" 0 < t < 1/2 we bave 
dH(q'((1 -- 2t)goH + 2tglH,),X) 
_< dg (((1 -- 2t)goA + 2tglA,) tJgoA),X) < a. 
Similarly, fol" 1/2 < t < 1 we have 
dH(q"((2t- 1)g,H, + (2- 2t)goH),X) 
<_ dH( ((2t -- 1)glH u+ (2- 2t)goH) LJ g, A),X) < a. 

Therefore, q(y) belongs to O(X,a). Since q(y)  Lo(n), we infer finally that 
q(y)  O(X a)  Lo(n), which completes the proof of the claire. 
Now assume that y  St(gH,AF(L/)) is an arbitrary point. We have to show 
that (y)  V(g, A). 
Let 
- = (goH),o,... , 
be the carrier of y, where go = g and Ao = A. Since goS, o ç ... ç g.,S,.  0, it 
follows from (5.1) that 

(5.2) gXx,  St(goSxo,L/) C V(go, Ao)/2 fol" all 0 < i < n. 
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Let V(g0, A()) t)(, the intersection Lo()nO(); e/2) for some Y e Lo(n). It follows 
from (5.2) that 

(5.3) 
Since 

for ail 0<i<n. 

it folh)ws from (5.3) that 
(5.4) 9iA, E O(t,e/2) for ail 0 < i < n 
By Lemma 4.4. R(y) belongs to the 1-skeleton of the -cell {90Ho,... ,9H,}, 
i.e., to the union 
Now, it follows from (5.4) and the above claire that the image under q of each 
l-cell (9, H,,.qjH), [)  i. j  , lies in ()( /2). Since 
¢'()= U q()' 
we infer that ¢(y) also lies in 
and )" is convex, the incquality d (conv A. conv B)  d(A, B)(sec [31, Theorem 
2.7.4(iv)]) implies that d((y),Y) < e/2, i.e., (y)  O();/2). Consequentl)ç 
 (v)  o(Y,e/2) Æ00,) = (0, 0) = v(, ), 
as required. 
The second claire of Lemma 5.2 now follows immediately from the first one. 
Indeed, for anv A G Lo(n) there is £ G A such that ç(.4) > 0. Then A  G(S), 
and hence, A  gS for somc g G G. It then follows from the definition of the map 
V: Æ0()  (U) tt (A)  «t(,(U)) ( t, vro« o a a.). y 

the first statemcnt, 4,(p(A)) e 
and 4, (p(A)) are e-close. This 
map of L0(n), which complctes 

V(g,A). Since A  gS C V(g,A), weseethat A 
means that t.he map (I)p is e-close to the identity 
the proof. [] 

Let (X, d) be a metric space with a geodesic (or convex) metric d, i.e., for anv 
two points x, y G X there is an isometry t : [0, d(x, y)] -- X such that t(0) --- x and 
t(d(x, y)) = y. For any element A G exp X the generalized closed r-ball centered 
at 4 is the set A = {x  X[ d(x,A) < r}. If X = l" and ,4 E B(n), then A is 
just the parallel body A + rB ', and hence, in this case, A is a compact, convex, 
centrally symmetric body. 
Lemma 5.3. Let ( X, d) be a metric space with a 9eodesic (or convex) metric d. 
Then for any two elements A. C  expX and any two nurabers r. s > O. the 
followin9 hold: 
(1) d.(A,.,B,.) < dH(A,B), 
(2) dHA,,,A) <_ If- s i . 
Proof. Since d is a convex metric, the first claire follows from [19, Proposition 10.5]. 
The second one follows from the property (Ap)q - Ap+q for any two nonnegative 
reals p and q (sec [25, p. 38, E×ercise 0.65.3(c)]). [] 
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Lemma 5.4. For each e > 0 there exist O(n)-equivariant maps fo, h«  Lo(n)  
Lo(n), e-close to the identity map of Lo(n), such that the images of f« and h« are 
disjoint. 

Pro@ Define a contilmous nlaI) 7 " Lo(n) --  by the rule 

7(A) = (1/2) min{dH(B,A),e} for every A  Lo(n). 

Let f be just the closed 7(A)-neighborhood of .4 in B , i.e., 
/«(A) = 
By the choice of 7(A), the set fo(A) is different from B ", and since A 
we sec that fo(A)  L0(n). It is clear from the construction that f« is a-close to 
the identity map of Lo(n). 
Let us check the continuity of f. ç have 
d,(f«(A),f«(C)) = d,(A(A).C(C))  d,(A,(A),A(c)) + d,(A(c).C,(c)). 
But by Lemma 5.3, 
d(A(A), A(c))  ]?(A) - ?(C)] and d(A(c), C(c))  d(A. C). 
Thus, 
dH(]«(A), fo(C))  b(A) - 7(C)1 + dH(A, C). 
Now the continuity of le follows from that of . The O(**)-equivariance of f is 
immediate from the invariance of the metric d. 
Next, we define the map he : L0(n)  L0() to be the composition p from 
Lemma 5.2. Then le(A) ¢ he(C) for all A, C  L0(n), since the comact set of fo(A) 
has a nonempty intcrior in the boundary sphere S -1 = OB  while the contact set 
of hh(C) has an empty interior in S - (this is because he(C)  ()). 

Lemma 5.5. There is an O(n)-equivariant strong deformation retraction (ft) o.f 
L(n) to its point B  such that .ft " L(n) -- L(n) is an O(n)-isovariant map .for all 
0 < t <_ 1. In particular, ri(A) # {B "} .for all 0 < t <_ 1. 

Proo.f. For each A  L(n) and 0 < t < 1. write 
.ft(A) = (1 - t)B n + tA. 

Proof of Theorem 1.3. Since, by Theorem 2.4. L() is an O(r)-AR, we infer that 
it is also an H-AR (sec, e.g., [30]). Therefore, by Theorem 2.7, L()/H is an AR. 
Since Lo(n)/H is an open subset of L(n)/H, it is a locally compact ANR. Now, in 
order to prove that Lo(n)/H is a Q-manifold it sutïîces, according to Toruficzyk's 
Characterization Theorem [28], to check that for every e > 0 there are continuous 
maps f', h'«  Lo(n)/H  Lo(n)/H, e-close to the identity map of Lo(n)/H, such 
that the images of .f' and h'« are disjoint. But this is immediate from Lemma 5.4, 
if we take for .f' and h'« the maps induced by f« and ho, respectively. 
The [0, 1)-stability of Lo(n)/H follows from Lemnm 5.5, which yields that the 
space Lo(n)/H possesses an obvious proper deformation (preimage of each compact 
set is compact) to infinity: 

(Lo(n)/H) x [0, 1)-- Lo(n)/H. 
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Hence, by a result of R. Y. T. \Vong [36], Lo(n)/H is honleOlnorphic to its product 
with the half-opCll interval [0, 1), i.e., Lo(n)/H is [0, 1)-stable. This completes the 
proof. 
The fol]owing lemlna for n = 2 was proved in [9]: 
Lemma 5.6. For each closed subgvu p K C O(n) actin 9 non-tunsitively on S -, 
and each e > O, there is a K-equivariant nzap h« : L(n)  Lo(n), e-close fo the 
identit n,ap of L(n). In particular, h(L(n)[Iç]) C L0(n)[K]. 
Proof. Let r : ()  L(n) be the O(n)-equivariant retraction froln Theorem 2.3. 
Because L(n) is «mpa«t (Theor«,nl 2.4), one Call find a real 0 <  < e/2 such that 
dn(r(A),,l) < e/2 for all A belonging to the 6-neighborhood of L(,) in (n), 
here du dcnotes the Hausdorff met.rie on (n). 
Fix a centrally symmetric, convex polyhedron P C  with a nonempty interior, 
inscribiug B , i.e., P C B" and all the vertices p,... ,p. of P lie on the unit sphere 
S n-1 = 0B n. Then the COIIVeX hull 
n : ,.,,nv (ç(p) ...  lç(p)) 
is a ccntrally symmetri«, compact, convcx, K-invariant subset of n. Since it 
contains , we ste that R bas a nonenlpty interior, and hence, R  B(n). XXç claire 
that tho boundarv OR does hot contain an (n - 1)-dilnensional elliptic domain. 
i.e., an open COlmected subsct of some (n - 1)-dimt, nsional ellipsoid surface lying 
in . It suffices to shov that none of the orbits 
(n - 1)-dilneliSiollal elliptic dOlnaill. Assume the contrary, that K(pi) contains an 
(n - 1)-dimensional elliptic donmin. Since K(pi) lies on the sphere S -1, then this 
domain should be in fact a dolnain of the sphere S -1. Since K(pi) is homogeneous 
and COlnpact, we conchlde that there are filfitelv many open subsets ç] ..... I of 
K(pi) such that K(pi) = I U.-. U I, where each i) is honieoInorphic to 1". Next, 
by the Domain Invariance Theorem (sec, e.g., [27, Ch. 4, Section 7, Theorem 16]), 
each ç should be open in the sphere S -1, and hence, the union Il U -.- U I- is 
open in S n-l. But 
Now, bv COlmectedness of S -1, it then follovs that K(pi) is the whole sphere 
S n-. Consequently, K acts transitively on the unit sphere S''-1, a contradiction. 
The clailn is proved. 
Now, let a be the distance of the origin from the boundary of R, and T = 
(1/a)R. Then T is a K-invariant, centrallv symmetric, colnpact, convex bodv that 
civumscribes the unit ball B , i.e., B  C T and the boundaries OT and OB  have 
a nonempty intersection. 
Setting 
'(A) = A (1- 6)T. 
we obtain a map h' : L(n)  B(n). Since T is a K-fixed point of B(n), we sec that 
h  is Iç-equivariant. 
Continuit.y of h  is evident. 
Since dn(A,A(1-3)B")  3 and A(1-3)B  C h'(A) C 4, we conclude that 
dn(A,h'(A))  6. In particular, h' is (e/2)-close to the inclusion L(n) 
We claire that h'(A)  hot an ellipsoid for each A 6 L(n). Indeed, if A C (1-3)T, 
then A # B , since T circulnscribes B  and 1-3 < 1. On the other hd. 
h'(A) = A in this ce, and hence, h'(A) is hot an ellipsoid. If A is hot contained 
in (1 - 3)T, then the boundarv of h(A) contains a domain lying in the boundary 
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of (1- 5)T. Since (1 -5)T = ((1 -5)/a)R and since the bouudary of R does uot 
contain an (n - 1)-diinensional elliptic doinain (as shown above), we conclude that 
the boundary of (1 - 5)T does hOt contain an (n - 1)-dimensional elliptic domain 
as well. Thus, h(A) is not an cllipsoid, and the claire is proved. 
Since r(h'(A)) and h'(A) bave the saine GL(n)-orbit, we conclude that r(h'(A)) 
# B' for each A  L(n). Since r is O(n)-equivariaut and h' is tç-equivariant, 
denoting by h« the composition rb', we obtain a K-equivariant nmp h« : L(n) -- 
Lo(n), E-close to the identity Inap of L(n). [] 

Proof of Theorem 1.4. For the first claire it sufiïces to show that Lo(n)/Iç is home- 
onlorphic to Q0, the Hilbert cube with a removed point. By Theorem 1.3, Lo(n)/K 
is a [0, 1)-stable Q-manifold. O,1 the othcr hand, Qo is a contractible [0, 1)-stable 
Q-mifold. Thercfore, according to a result of T. A. Chapman [12, Theorem 21.2], 
it remains only to check that Lo(n)/K and Q0 are homotopically equivalent, i.e., 
that Lo(n)/K is contractible. 
According to Theorem 2.4, L(n) is an O(n)-AR, which in turn implies that 
L(n)  K-AR (ste, e.g., [30]). Thcn, bv Theorem 2.7, L(n)/K is an AR, and 
hence, is contractiblc. I follows from Lemma 5.6 that the singular poire {B *} is 
a Z-set in the Iç-orbit space L(n)/K, and hence, according to [18], L(n)/K and 
Lo(n)/K bave the saine homotopy type. Since L(n)/K is contractible, we see that 
Lo(n)/K is contractible too. 
For the second claire, it suces to show that Lo(n)[K] is homeomorphic to Qo. 
First v show that Lo(n)[Iç] is a [0.1)-stable Q-manifold. Indeed. by Theorem 2.4, 
L(n) is an O(n)-AR, aud hence, L(n)[Iç] is an AR [2, Theorem 7]. It then follows 
that Lo(n)[K] is a locally compact ANR. Now by Toruficzyk's Caracterization 
Theorem [28], Lo(n)[K] is a Q-manifold if we observe that the equivariant maps le 
and h« from Lenmm 5.4 take Lo(n)[K] into itself. The [0.1}-stability of L0(n)[K] 
can be proved like that of Lo(n)/K in Theorem 1.3. Indeed, Lemnm 5.5 yields that 
the space Lo(n)[K] possesses an obvious proper deformation to infinity 
(Lo(n)[K]) x [0, 1) Lo(n)[tç]. 
Hence, by the result of R. Y. T. Wong [36], Lo(n)/H is [0, 1)-stable. 
Let us show that Lo(n)[K] is contractible. Since Lo(n)[K] ¢ , according to 
Lemma 5.6, the singular point { B  } is a Z-set in L()[K]. It then follows from [18] 
that L(n)[KI and L0(n)[K] have the saine homotopy type. But since L(n)[K] is 
an A, it is contractible, and hence, L0(n)[K] is contractible too. Since Q0 also is 
a [0, 1)-stable contractible Q-manifold, it only remains to apply the above-quoted 
result of T. A. Chapman to the Q-manifolds Lo(n)[K] and Q0. This completes the 
proof. 

6. PROOF OF THEOREM 1.6 

We first prove the following fact. 

Lemma 6.1. L0(n) and H0(n) bave the saine O(n)-homotopy type. 
Pro@ By [9, Lemma 4], there is an isovariant map f : Lo(n) -- Il(n), yielding that 
the image of f lies, in fact, in II0(n). Hence, the result follows froln the following: 
Claire. Every O(n)-equivariant nmp f : Lo(n) --* Ilo(n) is an O(n)-homotopy 
equivalence. 
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To prove this clailn we apply the ,lames-Segal Theorem 2.5. In our case G=O(n), 
T = L0(n) and Z = llu(n). 
By Theorem 2.4, L(n)  O(n)-AR, implying that L0('n)  O(n)-ANR. Let K C 
O(n) I,e a closed subgroup such that L0(n)[K] ¢ 0. As we have seen above in 
the proof of Theorem 1.4. L0(n)[] is contractible. Besides, it follows from the 
eq,fivarian«e of f that L0(n)[h'] C II0(n)[K], and so H0(n)[K] ¢ 0. 
On the other hand, since O(n.)/H  O(n)-ANR [26, p. 27], we infer that 
o,,,(o(,,)/) z o(,,)- ( [9. L,,,,,, 3]). Cosq,,.tb-, On) z O(.)- 
AR, i  1, and hence, Il(n) 60(n-AR, implying H0(n) 60(n)-ANR. Thus, it 
remains onlv to show that for each closed subgroup h" C O(n with H0(n)[K] # 0, 
Ilo(n)[tç] is contractibk, and, at the sa.me time, L0(n)[K] # 0. Ve will show that 
in fact l l(.n)[N] is a Hill,crt cube, implying thc contractibility of H0(n)[K]. Indeed, 
it is hot hard to sec that if II0(n)[K] # 0, thon thcre is an orbit type (H) such that 
(O(n)/H,)[tç] # O. This implies that (Iç)  (H,), and so, K C 9H,9 - for some 
9  O(n). But H, o«curs in L0(n)  a stabilizer. So there exists a body A  Lo(n) 
such that O(n)A = Hi. Conseqm'ntly. K C O(n)A- Since 9A  L0(n), we see that 
,,)[/ç] =  
s,, o()/,  O(,,)-NR,  se th«t (O(,,)/,)[ç] i a ,,o,,mpty N' Con- 
sequent ly, 
i a nondegçneate A. once, aecodili {o a esult of %si, [37], lira co/mtable 
product 
Q(H,)[Iç] = (Cone(O(,,)lHi)[tç])  
is a Hilbert cube. Tlfis in,plies tiret N (ç(Hd[K]) is a Hilbert cube, and hence, 
i=1 
N0(n)[h] is contractible. By applying the above-mentiolwd James-Segal Theorem, 
we complete the proof. 

Proof of Theorem 1.6. Since by Theoreln 1.3, Lo(n)/H is a [0, 1)-stable Q-manifold, 
according to Chapman's theorem [12, Theoreln 21.2], it remains only to prove that 
FIo(n)/H is a [0, 1)-stable Q-manifold of t.he saine holnotopy type as Lo(n)/H. 
The fact that FI0(n)/H is a Q-manifold is proved in [9, Theorem Al]. Its [0, l)- 
stability follows from Wong's theorem [36] if we observe that FI0(n)/H possesses a 
proper deformation to infinity: 

(1-Io(n)/H) x [0, 1) -- IIo(n)/H. 

Indeed, this follows easily from the conic structure of Il(n). 
Finally, that II0(n)/H and Lo(n)/H bave the saine homotopy type follows iln- 
mediately from Lemma 6.1. 

7. PROOF OF THEOREM 1.9 AND PtEDUCTION OF CONJECTURE 
V'e start with the following lemma. 
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Lemma 7.1. Let G be a compact group. N C G a closcd, normal subgroup and X 
a G-ANR (rcsp., a G-AR). Then the N-fixed point set X[N] is a G-ANR (resp., a 
G-AR) as well. 

Pro@ According to [5, Corollary 5], there is a normed linear space L such that 
X can be embedded as a closed invariant subspace into Z = C(G, L), the normed 
linear G-space of all continuous maps f: G -- L endowed with the sup-norm and 
with the action gf of G defined by 
(gf)(x) = f(xg); f E C(G,L) g,x  G. 
Then there is a G-retraction r: U -- X for some open G-neighborhood U of X in 
Z (resp., U = Z). Therefore, it sutïîees to prove that Z[N] is a G-AR. One easily 
secs that Z[N] = C(G/N,L), where the G-action g on C(G/N,L) is defined by 
(gch)(xN) = ch(xgN), for ¢  C(G/N,L) and 9  G, xN  G/N. 
Now C(G/N, L) is a G-AR by [5, Theorem 8]. [] 
Let sexp S n be the subspace of exp S n consisting of all centrally symmetric sets 
A C S', i.e., A = -A. By sexp0 S' we will denote the complement (sexp S')\{S'}. 
Evidently, sexp S' is an O()-invariant subset of expS'L 
The next lemma is ilnmediate froln Theorem 3.1 and Lenmm 7.1 if we observe 
that sexpSn-1 : (exp Sn-1)[N] with N : {ln,-l}: 
Lemma 7.2. sexpS n-1 is an O(n)-AR. 
Lemma 7.3. Thcre exists an O(n)-cquivariant map f : sexP0 S n-1 -- Lo(n). 
Proof. For every A G sexp0 S'-1 , let 
(A) : conv(A O B(O, 1/2)), 
where B(0, 1/2) is the elosed 1/2-ball in ]W  eentered at the origin, and conv stands 
for the convex hull. Clearly, ç is a well-defined, continuous, O(n)-equivariant map 
of sexp0 S'-1 into B(n). Furthermore, ç(A) is hot an ellipsoid beeause A -¢ S'-1, 
implies that the boundary O(ç(A)) contains a nontrivial line segment. Now we 
set f = r, where r : B()  L(n) is the O(n)-equivariant map from Theorem 
2.3. Sinee ç(A) is not an ellipsoid, and since r preserves the GL(n)-orbit, we 
conclude that f(A) is hot the unit ball B'; so f(A)  Lo(n). Since r and ç are 
O(n)-equivariant, so is f. [] 
Lemma 7.4. Let Iç C O(n) be a closed subgroup. Then the following conditions 
are equivalent: 
(1) Iç acts non-transitively on the sphere S'-1. 
(2) (expo S'-l)[Iç] # O, 
(3) (sexp0 S'-I)[K] ¢ 0, 
(4) Lo(n)[K] ¢ O. 
Proof. (1)  (2). If K acts non-transitively, then there is a K-invariant proper 
subset A C S'-1; so A  (expo S'-I)[K]. 
(2)  (3). If A G (exp 0 Sn-1)[K], then either 
(-A) U A  S '- or (-A) fq A  {3, 
and so at least one of the sers (-A)t2 A and (-A)ç A belongs to (sexp0 S'-) [K]. 
Thus, (sexp0 S'-I)[K] ¢ !3. 
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(3) == (4) is ilmnediate flore Lemma 7.3. 
(4) = (1). If there is a body A  L0(n)[h'], hen the conter set A S - is a 
nonempty, K-invariant, proper sui»set of S -1. So the action of h on S - is not 
tra.nsit ive. 
Lemma 7.5. Every O(n)-equivariant nap f : sexpo S -1  Lo(n) is an O(n)- 
homotopy equivalence. 
Proof.  are going to apply thc James-Segal Theorem 2.5 with G=O(n), T 
sexpo S - and Z = Lo(n). It follows from Lcmma 7.2 that sexpo S -  O(n)- 
ANR. Since L()  O()-AR (sce Theorcm 2.4), we see that also Lo(n)  O(n)- 
ANR. 
Let h- C O(n} be a closed subgroup. Then by Lelmna 7.4, (sexpo sn-)[h 
ifl" Lo(,,)[tç]  0. Lct Lo()[tç]  0 and [ç'= [ç x {1,-1}. Sin«e 
(exp o S-')[Iç '] = (sexp 
the action of K' (m S - is hot transitive, i.e., Sn-/K ' is hot a singleton. Since 
(ex I, S n1 ) [/ç'] is h()llle()lll()rphic fo exp(S -1//ç') and S - / K' is a llondegeerate 
Peano COlRimlmn, by the Çurlis-S«hori->sl Hyperspace Theorem (see. e.g., 
8.4]), exp(S-/tç'), and he,,«e (sexpS'-)[lçl, is a Hilbert cube. Çonsequently, 
(sexp0 S-)[K], being a Hilbert cube with a relnoved point, is contractible. 
According to Lemlna 5.6. the sil,gular point {B"} is a Z-set in L(,,)[h']. Since 
L(n)[K]  AR, it then follows from [18] tha L(n)[tç] and L0(n)[K] have the saine 
holnotopy type, and hence, L0(n)[K] is contrtible too. Now, by applying the 
above mentioned .lames-Segal Theorem 2.5, we complete the proof. 
Similarly, the following can be proved: 
Lemma 7.6. Every O(n)-equivariant map f : sexp0 S -  exp0 S - is an 
O(n)-homotopy equivalence. 
Proof. By Lemma 7.4, for any closed subgroup K C O() one h 
(xp0 s-)[h ] # 0  (ep0 s-)[h ] # 0. 
As in the proof of Lelllln& 7.5, (sexP0 S-)[K],  well  (exp0 S-)[K], is 
contractible whenever (exp0 s-l)[lç] # 0 (or equivalently, (sexp0S-)[K] # 0). 
Now apply the .]ames-Segal Theorem 2.5. 
Lemma 7.3 and Lcmma 7.5 have the following immediate consequence. 
Corollary 7.7. sexp0 S -1 and Lo() bave the saine O(n)-homotopy type. 
Analogously, Lemma 7.6 implies the following. 
Corollary 7.8. The natural inclusion sexP0 S -1  exp0 S - is an O(n)-homo- 
topy equivalence. 
Proof of Theorem 1.9. Immediate from Corollaries 7.7 and 7.8. 
Our final result reduces Cnjecture 1.8 to an eier one: 
Theorem 7.9. For each closed subgroup H C O(n), the two H-orbit spaces 
Lo(n)l and (exp0 S-)l am omeomo,ic iff (exp 0 S-)/ is a Q-mai- 
foZg. 

For the proof we need the following fact. 
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Lemma 7.10. There exists an O(n)-equivariant strong deformation retraction 
of expS '-1 to its O(n)-fix«d point {S '-1} such that ff(A) 7  {S '-1} for ail 0 < 
t <_ 1 and A  expo 

Pro@ Observe that the usual spherical nmtric d 011 S n-1 is O(n)-invariant and 
convex. For each A  exp S'-1 and 0 < t < 1, write 
ft(A) = {x  S n-1 I d(z,A) <_ (1 - t)dH(Sn-l,A)}. 
Due to the convexity of d this homotopy is continuous; it is also equivariant, since 
d and dH are iuvariant. Other required properties of (ft) are evident. [] 

Proof of Theorem 7.9. Since by Theorem 1.3, Lo(n)/H is a Q-manifold, only the 
"if" part requires a proof. 
So, assume that (exp0 Sn-1)/H is a Q-manifold. It follows from Theorem 1.9 
that 
(exPoSn-l)/H and Lo(n)/H 
bave the salue homotopy type. Moreover, by Theorem 1.3, Lo(n)/H is a [0, 1)- 
stable Q-manifold. Thereforc, according to Chapmm's theorem [12, Theorem 21.2], 
it only reluains to see that (exp0 S-I)/H is [0.1)-stable too. Indeed, Lernrna 7.10 
yields that (exp0 S'-I)/H possesses an obvious proper deformation to infinity: 
((exPo5'n-1)/H) x [0,1) -- (expo,.g'-)/H. 
Now Wong's result [36] implies that (exp0 S'-I)/H is [0, 1)-stable, and this com- 
pletes the proof. 

Acknowledgement. The author is grat.eful to the referee for helpful suggestions 
and remarks which led to a number of improvements of the text. 
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LOCAL SOLVABILITY AND HYPOELLIPTICITY 
FOR SEMILINEAR ANISOTROPIC 
PARTIAL DIFFERENTIAL EQUATIONS 

GIUSEPPE DE DONNO AND ALESSANDRO OLIARO 

ABSTRACT. We propose a unified approach, based on methods from microlocal 
analysis, for characterizing the local solvability and hypoellipticity in OEc and 
Gevrey G a classes of 2-variable semilinear anisotropic partial differential oper- 
ators with multiple characteristics. The conditions imposed on the lower-order 
terres of the linear part of lhe operat.or are optimal. 

1. INTRODUCTION 

We consider a class of selnililmar anisotropic equations with multiple characteris- 
tics in two variables (x, y) belonging to the set fl := { x = + : < 5}, 5 suciently 
small, of the form 
"  = f(x, 
(1.1) P(x, y, D, D)u + F(x y, 
where the linear part is given by 
m l j 
(1.2) P(x,y.D.,D) = D -bo(x,y)D +  aO(«,y)DD , 
 -i, D -i " we 
with m,d,j, 1 Ç Z+, 0 < t < ,  sucientlv small, Dz = = o, 
 DD, and so the nonlinearity 
shall also sw that la +j is the anisotropic order of  J 
involves derivatives of anisotropic order less than m- t. Our main aire is to propose 
a unified approach for a complete analysis of t he influeuce of the lower-order terres 
of (1.2) on the solvability and hypoellipticity of (1.1) in the C  category and in 
the Gevrey spaces G  beyond the critical index m/(m- 1). The arguments in our 
proofs are baoed mainly on microlocal tools: pseudo-differential operators, wave 
front, sets, allowing relevant simplifications in the study, and Sa tedmiques. 
Some papers have been devoted to the study of this kind of probletn; see, for 
example, Hounie-SatUiago [HS] and Gramchev-Popivanov [GP] on the local solv- 
ability of semilinear parti differential equations in the case of simple characteris- 
tics, Gramchev-Rodino [GR] about Gevrey solvability for equations with multiple 
characteristics (see also Spagnolo [SP] and Kajitani-Spagnolo [KS]), and Garello [G] 
regarding the inhomogeneous elliptic case; see also gananin [SI on the C  local solv- 
ability of equations of qui-principal type and Lorenz [LOI regarding anisotropic 
operators with characteristics of constant multiplicity. 
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We consider an F that is C  and nonlinear, and we assume that the coefficients 
in (1.2) are C oe. We always suppose that 

(1.3) 

d < m, ,b0(0,0) ¢ 0, F(x,y,O) = O. 

We recall that the nonzero requirement on b0 is an invariant nondegeneracy condi- 
tion, usually required in the study of the local solvability and hypoellipticity of the 
linear operator (1.2) in C  and in the Gevrey classes G «, er > --1, see for example 
Liess-Rodino [LR2], De Donno-Rodino [DR2], in which Gevrey hypoellipticity for 
PDEs with high nmltiplicity is proved. Let us also observe that if bo(x, y) ¢ O, 
then the operator is quasi-elliptic; thc result.s of hypoellipticity and local solvability 
are well known in this case. Regarding G « data, sec for exmnple Marcolongo-Oliaro 
[MOI, in which the local solvability is proved in the n-dimensional case and under 
hypothcses on the quasi-principal symbol; in the prescrit paper we admit less reg- 
ular data f(x, y) with respect, to the case studied in [MOI, but we add hypotheses 
on the lowcr-order terres. In this frmne it will be convenient to use the Sobolev 
anisotropic space Hî, q _> 1. defined by 
q 

A l÷q H is an algebra; cf. 
f(, l) being the Fourier transform of f(x, y). For s > ----, _ 
q 
the inhomogeneous Schauder estinmtes and Garello [G, Proposition 2.5]. Moreover, 
we define the anisotropic characteristic manifold 

(1.4) 

r:= {(z,y.f,.) e  x t \ {0}..r-,b0fe = 0}. 

We recall the definition of the Gevrey anisotropic space G(q"q)(fi). 

Definition 1.1. Let q > 1, q2 > 1. We denote by G(q'q)(fi) the set of all 
the flmctions f  C() such that the following condition holds: for every com- 
pact K C [2 there exists a positive constant Ch- such that suPK ]OtOJyf(X, y)] < 
C+.Y+l(l!)q (j!)q for every l,j  Z+. 
As usual, G(oq'q)(f2) is the set of all the functions in G(q'q)() with compact 
support in [2. 

Let us state the nmin results. 

Theorem 1.1. Let (/*,j*) e Z_ be the unique couple in Z+ having anisotropic 
" = l* m j. ml* + dj* < dru. We assume that for 
order -x ê + , with d(m- ½ ) < k* = 
(x,,,) e  

(1.5) 

i)  at.. (x, u) ¢ o, 
ii) for all (l, j) such that dj + ml > k*, 

 al*. (x, y)c alj(X, y)l+l*TJ+J* OE O, 

iii)  al.j.(X, y) bo(X, y)d+l*TJ"  O. 
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Choose and riz t =.m- and s > 2  + . Then we can find 50 > O, depending 
on P and s, such tlat for every f E HS (Q) witl compact support in Q the equation 
(1.1) admits a solution u  H +'-" with r* = m - @, provided # = 1 if F =- 0 
tri 
and P.]If[[Hç < P.o for some 0 < Po « 1 dependin 9 on the nonlinear terre F if 
F  O. Finally. the linear operator P in (1.2) is COe(Q) hypoelliptic, and if 
coefficients are analytic, P is G°(Q) hypoelliptic, a > d i.e. if u is a 
' -- k*--d(rn--1) ' ' 
distribution in ft such that Pu  G°([), then u  

Remark 1.1. The conditions (1.5) in TheoreIn 1.1 could be illustrated in a sim- 
pler way, observing that actually one estimates the imaginary part of the symbol 
}-<+j<m aq(x, y) l j in the corresponding oI)erator (1.2) on the quasi-conic 
characteristic set !  -.} bo d = O. e.g., substituting c = ( b0)-} r/. the condition 

(1.5) reads 

(1.6) I'(<,+< (b°)£lj) _ Z alj Il  

with k* = ml* ÷ dj*. 
Theorem 1.1. 

OE Fit/[ -a-, ri » 1. 

This clarifies, at least intuitively, the loss of derivatives -- in 

Rernark 1.2. It is always possible to ret)hrase the previous assulnptions in Theorem 
1.1 directly on the coetï-icients of P. For example, if .Rb0 > 0 and m. d are o(ld, the 
conditions i), ii), iii) are respectively equivalent to: 
if) al.j.(x.y) > 0 (<0); 
il t) for all (/,j) su«h that dj + ml > k* = dj* + ml*, ,az(x,y) >_ 0(_< 0) for 
j + j* and l + l* both even or both odd, and  au(x, y) = 0 otherwise; 
iii )  bo(x, y) <_ 0 (_> 0) for j* and d+l* both even or both odd, and  b0(x, y) = 
0 otherwise. 

In the picture on the next page, which resenlbles the Newton polygon pictures, 
we show the geometrical memling of hypothesis il) in Theorenl 1.1 (or equivalently 
assumption ii ) in Remark 1.2). We consider the operator of order m = 9 with 
d = 7: 
_hr3r5 " 6 (X, l j 
9 _ (l - iy2)D + y 
(1.7) Du 
y)DD, 
9 - 61 
7/+3< 
is C  and Gevrey hypoelliptic and C  solvable bv Theorem 1.1. 
 want to sudy now the case in which hypothesis i) in Theorem 1.1 is hot 
safisfied: the basic idea is to refer to the Gevrey cluses and ransform tire operaor 
P in (1.2) ino another operaor that safisfies i. To flfis aire, we introduce he 
anisotropic Gevrey-Sobolev spaces ,q,( 
fimctions for which 
(1.8) IIf]l;:$, 
where q OE i is the Gevrey order, s > 0 the Sobolev index, and we take r  (0, I), 
 » O;  = (y, ) is a nolmegative function belonging to the H6rmander class 
sLo((-, )  ). 
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Dy 

9 

6 

5 

To every couple (D,D) = (l,j) with 91 + ïj < ï- 9, 
t j 
corresponds a mixed derivative DxD in (1.7) having 
analytic coefficient atj(x, y). In the hypoellipticity 
region 7(9 - !) < 91 + 7j < î- 9 there are three 
candidates to give anisotropic order * * 
We have (l*,j*) = (6, 1), a6(x,y) > 0 and 
9-6+7:=k*=61. For (1,j)=(3,5), 
3.9+5.7 = 62>k*, a3(x,y)0. 
The anisotropic order of the other 
I j 91+73 
mixed derivatives DD is d , 
less than , so we do hot 
require any sumptions. 
b0 > 0, b0  0. 

2 3 6 7 

Theorem 1.2. In the equation (1.1) let t]e datum f be in "o('q:)(), r C (½, 1). 
q2 > 1. Fix t > 1 - r and assume that the coefjïcients of P in (1.2) are analytic 
and for (x, y, , U)  E one of the following conditions holds: 
lai aj(x,y)] j+-I  0 ( O)for dj + ml > (- t)d. and, moreover, 
bo(x ,y)dr]m-1  0 ( 0): 
[b] alj(X, y)(8"ig )l?]j+m--1  0 ( O) for ai+ml > (m-t)d, and bo(x, g) 
(siÇlt )d.i]m-1  0 ( 0). 
Assume moreover that the nonlinear terre F is analytic with respect to (x. y). entire 
.I.j 
with respect to 00, and (1.3) holds. Then. fixing  and taking s large, we can 
find 5o > O, depending o P and s, such that for every f  H¢,«() with compact 
+-(1-),o  
support in  the semilinear equation (1.1) admits a solution u 
pïoid«d  =  g   o. ad Hf[[.:, <  fo o« o < # «  d«p«di o 
the nonlinear terre F  F  O. 

The study of the weakly hyperbolic equation (1.1) resembles the study of degen- 
erate parabolic equations, in that mder suitable hypotheses equation (1.1) behaves 
like them in regard to local solvability and hypoellipticity in the C  cat.egory and in 
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m See, for example, Gramchev-Popivanov-Yoshiuo 
some Gevrey spaces G «,  > --2" 
[GPY, Section 3]. 
In Theorem 1.2 we require that thc nonlinearity is analytic with respect to 
t j 
(x, g, OxOu). One encounters highly nontrivial diflïcultics in getting hard anal- 
ysis type estimates on composition of nonanalytic Gevrey nonlinearitics, and in 
the paper Gramchev-Rodino [GR] differcnt Gevrey norms involving power series of 
finite Sobolev-type norms are used. A possible analogue of Theorem 1.2 could be 
proved in the anisotropic case for nonanalytic Gevrey nonlinearities, and this will 
be thc subjcct of a future paper. 
We point out that in hypotheses [ai and [b] of Theorem 1.2 thc exponcnt m - 1 
plays the role of j* in assumptions il) and iii) of Theorem 1.1. In fact, for a suitable 
z 
fixed ¢(y, ç) E Sï, o((-6, 6) x IR), the operator 

/ := er(y'D)p(x, y, Dx, Du)e -r(y'D) 

contains ail the terres of P and an additional pseudo-differential terre 
N(x,y)lD[ -,r) ,,-1 

where Nf(x, y) satisfies condition i) in Theorem 1.1. 
Observe that for the existence of/.j E Z+ such that m,-t < i +j < ru, t < ½, 
we bave to require m _> 4. The theoreln applies, of course, a.lso ils the case when 
the assumption on the at a- is empty; in this case we recapture the result of [lklO]. 
Passilag to the standard isotropic Gevrey classes G«(l), defined by the estimates 

sup lOtOf(m, y)l < c+J+'(i!j!) «, 
K 

K CC çt, 

m 
we may conclude that equatiou (1.1) is locallv solvable for f  Gô (ç/). 
We observe that under assumption i) in Theoreln 1.1 we obtain solvability in 
C , as well as C  hypoellipticity, while for the operator / and Theorem 1.2 we 
only get local solvability in G. 

Remark 1.3. The meauing of conditions la] and [bi in terres of thc imaginary part 
of the coeflïcients of P depends on ru, d, and the sign of b0(0, 0). Let us analyze 
some situations. 
 If m and d are even and .b0(0.0) > 0, we can treat equation (1.1) in one 
of the following cases: 
(a) ,bo = O, ,a 0 <_ 0 (>_ 0) for l even and j odd, ,a O = 0 otherwise; 
(b) b0 = 0, ,a 0 < 0 (> 0) for l and j odd, ,aj = 0 otherwise. 
 If m is even, d is odd and .b0(0, 0) > 0, since p(x, y, ç, r]) is quasi-elliptic for 
ç < 0, the hypotheses of Theorem 1.2 on the linear part become b0 = 0. 
C3a 0 < 0 (> 0) for j odd, a 0 = 0 for j even. 
We also observe that if m and d are even and .b0(0, 0) < 0, the operator is quasi- 
elliptic and we may apply known results; cf. Mascarello-Rodino [MR] and Rodino 
Let us compare our result with the previously known ones. For the sake of 
brevity we limit our attention to a model of the form (1.2) with d = m - 1. where 



3410 GIUSEPPE DE DONNO AND ALESSANDRO OLIARO 

we fix attention Oll the case x = 0, V = 0: 

(1.9) 
D 

 r2 rm--3 
- D - + iavD -1 + i(]1 Yq -[- 2) Dr D' -2 + 
 D3D m-4 
+ i(y2 +  2)  . , a,,2,,p.2   ('  3). 
Let us ana]yze first the case 2 = 7 = P = p2 = 0. For a =  = 0 the operator 
(1.9) is hot hypoelliptic; observe also that for a # 0, ç = 0, h = 1, (1.9) is hot. 
hypoelliptic and hot locally solvable, cf. Corli [CI. For  # 0, fl = 0, h even, 
(1.9) is hypoe]liptic and locally solvable, dcspit.e the fact that bo(O,O) = 0, cf. 
Menikoff [MI, Popivanov [Pli, Roberts [RI; for both ,1 # 0 and h even, the 
operator (1.9) is hot hypoelliptic if h is sufficiently large with respect, to q  1, cri 
Pupivanov-Popov [PP], Popivanov [P2], while for h _< q- it is hypoelliptic and 
locally solvable, cf. Cramchev-Popivauov [CP1]. 
Theorem 1.1 gives new conditions on the coefficients of the terres D D n-j-1 
for models of the type (1.2), to guarantee hypoellipticity; so now we discuss the 
case when 2, 7, p, p, are hot ail zero. 
Let us observe that, if j* is odd, thon (ii), (iii) in Theorem 1.1 actual]v imply 
atj(x,)  O and bo(x,)  O for cven j < j* (d=m-limpliesj+l=j*+ 
l* = m-1); as examples of hypoel]iptic and locally solvable operators characterized 
by Theorem 1.1, consider in this case (1.9) with a = l = 0, 2 # 0 (j* = 1), and 
(1.9) with  = 2 = 7 =  = 0, 2 > 0, q even (j* = 3), having the saine 
b0 as the non-hypoel]iptic operator D  - D -. If j* is even, then (il) implies 
a 0  0 for odd j < j*; as a corresponding exa.mple of hypoel]iptic and locally 
solvable operator consider (1.9) with  = ff2 = 0, a7 > 0, h even (j* = 2). The 
order  has to be chosen sufficient]v large to satisy, the assumption - > j*. 
Now we discuss for the preceding examples the problem of local solvability in 
t.erms of Cevrey classes, arguing in the isotropic spaces G«(). Concerning (1.9) 
with h even, to which we may add arbitrary perturbations of lower anisotropic 
order, we bave -local solvability for  <  This follows from Theorem 1 o and 
is also a consequence of Marcolongo-Oliaro [MOI. The result is sharp, in the sense 
that when a # 0 and h = 1 (1.9) is hot «-locally solvable for  >  
-2" sec Corli 
ICI. For , l # 0, 2 = 0, it was proved by Popivanov-Popov [PP] and Popivanov 
[P2] that (1.9) is hot C  local]y solvable, and receut.]y for h even, Marcolongo 
[MAI ext.ended the result to -non-solvability for  >  +e(h) with e(h)  0 for 
h  oe. As new applications of out result.s in terres of G « ]ocally solvable operators 
characterized by Theorem 1.2, consider (1.9) with  # 0, a =  = 0, q even (m OE 
4), which is  locally solvable for  <  independentlv of lower anisotropic order 
perturbations. Compare in particular with D v TM - D - + DuD. -2, for which 
the change of sign of t.he imaginary part of the coefficient gives -non-solvability 
for  >   have no change of sign and -solvabilflv also in the int.erval 
2" 
 >>  Moreover, let us consider (1.9) with >0,=2=2=0, 
--3 --2 " 
q even (m > 10), which is  locally solvable for  <  The addition of lower- 
order anisotropic t.erms might produce non-solvability phenomena in C  and G « 
for large . Observe finally that if the imaginary parts of a 0 in (1.2) vanish of 
high order at t.he origin, then t.he ]ower-order terres have no influence on the local 
solvability. As an example we consider the operator 
 iv)D + even, 
(l.lO) D - (1 - iy'DD, k 
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which is C  locally solvable and C  hypoelliptic foi" k < l; see the arguments in 
the book by Gralnchev and Popivanov [GP1, Theoreln 3.1 and Chapter 4], and see 
also Theorem 3.1 in Gramchev [GG]. Vë also observe that by applying Theoreln 
4l, too. 
1.2 we obtain Gevrey G « local solvability, a < , for k >  

2. GEVREY-SOBOLEV SPACES 

As a preparation fol" the proof of Theoreln 1.2, in this section we studv a class of 
Gevrey-Sobolev spaces defined on the strip IR x (-& 5), 5 > 0. These spaces have 
been intl'oduced in the n-dilnensiolml case in [lO]; here we prove some results that 
will be used in the next sections for the local solvability of (1.1). 
s, (_, 
Definition 2.1.  define the Gevrey-Sobolev space .q«( x the set 
 we sllall 
of ail functions f G L( x (-&5)) such that (1.8) holds. Yiting p = q, 
say that ¢(y, ) is a weight fimction of order (r, p). The operator e ¢(u'D*) acts on 
the function f in the following wav: 
whcre f(y, ) = j'e-¢f(x, y)dx, and H(  (-& 5)), for s intcgcr, is the space 
of all g(x,y)  L( x (-5,5)) such that 
the definition of H( x (-5, 5)) cxtends to ccry s > 0 bv interpolation. 
Reark 2.1. The operator e r(y'D) and its inverse e -r(y'D*) establish an isometrv 
  (-, )) ,, n; (  (-, )). 
between the Hilbert spaces '.q,( 
 need to introduce suitable equivalcnt norms on the spaces 
.«, (  (-, )). 
First we recall that the following idcntities hold: 
(2.2) 
h=0 

(2.3) 

: Z -(J) ' i 
"-h'(Y" D«)Dh e-'(u'D), 

h t __O 

(2.4) 
for every k, 3 
q(J) rg De) and 
j--lk ' 
In particular, we obtain that 
q(d)(v,D) : 1, 

and 

Dkxer¢(Y.D*) = er¢(Y,D,) Dkx 
G Z+, where the symbols of the pseudo-differential operators 
r ()j_t(y," Dx) belong to the H6rmander class S, op(-O ((-5, 

qJ)(y,D«) = j-(Du¢)(y.D), 

r(oJ)(y.D«) = 1, rJ)(y,D«) =-j'r(Dv¢)(y,D). 
For dctails, see [MO, Lemma 1.1], in which (2.2)-(2.4) are provcd in the n-dilnen- 
sional case. 
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Definition 2.2. We sT,' that a function f belongs to the space H 
(m ,p2) - 
0 < Pl  1, 0 < P2 <_ 1, if and only if f  L2(N 2) and 
(2.5) Iii(x, )IIH.=,() -- (1 + + 
Lemma 2.1. Let f  Hftm,v:)(N2). s _> O; let t and h be such that t +__v:h _< s. 
Then 
u,« ()  ()' « th« p«eo-«ig«ti op«to ith o 
ç', respectively; we write (, q) for the dual variables of (x, y). 
ool. Lt ç = [-, ]  [-1, 1]; w 
t h 2 
 II<D)SlI()+ II(DF'SlI() + IISlI(). 

Remark 2.2. Let us suppose that Pl. p2 E Q, and let s be a positive integer such 
that pis is an integer, j = 1,2. By Lemma 2.1 we have that an equivalent norm in 
Hp,p2 ) (N u) is given bv the following expression: 
(2.6) IIDP/IIL2(2) + IID'flIL() + 11/11(2). 
Lemma 2.2. Consider u  H( × (-)) with p a rational number and s a posi- 
tive integer such that ps is an integer; we write 0 for IR x (-5, 5). Then there exists 
a function ls,oU  H  (]R 2) that extends u. Moreover, we can find a constant C 
(p,) 
such that: 

(2.7) 

_ ps 
11gF,,oull() < cI19 ullL(×(-.)), 

IIDt,,oll(2) _< CIID-II(×(-,)). 
This lemma can be proved using an argmnent similar to the one developed in the 
isotropic ce by Egorov and Schulze in [ES, Theorem 27]. The proof is omitted. 
Remark 2.3. Observe that [[U[[L(x(-&))  [[/s,Ou[[g(e:), fs,Ou being an extension 
of u; so [[U[L(ex(-,)) and [[/,ou[[g:(e) are equivalent. In the saine way we obtain 
ps s  ps 
that D U]L(ax(-6,6)) and ]DU]L (ax(_6,6)) are equivalent to D l,OU[L:(a:) 
and Dl,oUL(a:) respectively. So if s is an integer such that ps is an integer 
and f  H(  (-&5)), using Lemma 2.2 and Remark 2.2 we can prove that an 
equivalent norm in the space H(  (-5, 5)) is given by the following expression: 
(2.8) IID fll((-,)) + IIDfll((-,)) + Ilfll((-,)). 
Theorem 2.1. Let us fix s > O, r > O, r  (0, 1], p E (0, 1} and assume that s is a 
positive integer such that ps is an integer. Then the following norms are equivalent: 
(a) f ¢:.((_)); 



SOLVABILITY AND HYPOELLIPTICITY FOR SEIvlILINEAR PDES 3413 

(c) 
(d) 

Pro@ 

On tha other hand. sinca f :«((-aa)) S ]]l,o(e('D):)l]:,)(), b e- 
mark 2.2 and Lemnm 2.2 wa hava 
 +j 5s 
(b)  (c). Using Lenmm 2.1, Lemma 2.2, Relnark 2.3 and the identities (2.'2)- 
(2.4), we obtain 
 I1: ¢(') 
 +j5s 
J 
} +js h=O 
+3s h=0 k=0 
 Ç (IIDFl,o: ('D /llz(e) + IIDl,oe('D  
+ II/.o: (') 
 +j<s 
 - 

In the opposite direction we moEv use similar arguinents. 
we have (c)  (d). 

By the saine arguinents 

Now we prove some important results that will be used in Section 3 for the 
solvability of the seinilinear equation. 
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(o   and 
Theorem 2.2. Let O < p < 1 and O < r < 1. Let f 
q2 > 1. Then for all s > 0 and for everg weight function ¢(y,) of order (r,p) 
s,¢ 1 
there exists ç > 0 such that f  ,q,, where q = . 
s,¢ 
Pro@ First we observe that ,, C ;'. for s > t: then, without loss of gen- 
erality, we mav assume that s is a positive integer such that ps is an integer. By 
Theorem 2.1 we can write 
The fimçtions DS, f(x, y) obviouslv belong to G ç'q2) (  (--, )) for every mutti- 
index  = (,2) suçh that  + 2 < s with the saine constant Ch"  in 
p 
Definition 1.1, depcnding on s. For evcry integer k, applying the Fourier transform 
with respect to x we can find b   depcnding on supp f, such that 
It tbllows imme,liately that (1 + Içl)lf()l  C(C()) 
where we suppose C  1. So we bave 
1 1 + Içl »  1 
2Cx - 
k k 

therefore we obtain 

Remark 2.4. Let 0 < p < 1 and 0 < r < 1. Let f  G(oql"q2)(]× (-5,5)) with 
1 < ql < p and q2 > 1. Then, for every s > 0, for every weight fimction ¢(y, ) of 
'¢ (-5, 5)), where q -- . 
order (r, p) and for every r > 0, we bave that f  ,q« ( x -  
Theorem 2.3. Let (y. ç) be essentially subadditive with respect fo , i.e., 
4 [G]. Let so   satis/y the assumptions o/Theorem 2.1. Then for eveW 
s > So the space '¢' (-5,)) is an 
,q«(  algebra, and there exists a constant Cs 
such that 

(2.10) 

Pro@ We begin by proving Theorem 2.3 with s a positive integer such that ps is 
an integer. Using Theorenl 2.1, (c) and applying the Leibniz rule, we obtain 
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where uf(x,y) = er¢(u'D*)Dt3u, v.r(x,y ) = e-¢(u,D*)D'rv and the norms are in 
IR × (-5, 5). Applying the Fom'ier transform with respect to x and writing *(O for 
the convolution in the ç variable, we bave 

IIvllr¢:.  <c , ï 

iIT<u.«) ((-T(u,o) ,(«) (-(u,o))(, OIIL ' 

where f and 7 stand for the partial Fourier transform with respect to x of uf 
and v. r respectively. The flmction b(y, ) 1)eing esscntially subadditive with respect 
to ç, it follows innnediatelv that e -(u'()--(u'ç-u)--'(u'u) <_ e C, and so 

Since we have required s > -p, at least one of the inequalities --p + ri2 < s - -- 
and "rp + 72 < s - -¢- must be satisfied. Let p > 0 |)(' Sll('h that 

 --+o2<8 } 
p 4p 
p 

Using Lelnma 2.2 and Yollllg's estimatcs, we have 

+C3 

p+l 

4t» 

- +2_<s 
p 
1+2<s-- 4P+pl--Pp 

p+l 

4p 

s, 
Since , v  ]HI_,q.r(N x (-5, 5)) aud  +ri2 < s,  +72 < s, Theorem 2.1 and (2.7) 
assure us that ]](lo,oUO)]]L(a )  I111:. and ]](lo,oV)]]L(a ) 
Moreover, if  P+ 
 + 2 N s p p, by the H61der inequality and (2.7) we have 
IIq  +..o)(ç, ) IIL, () 
 C41I( + I1 ç(+) + I1 `» )(+.) IIL() 
The saine argmnents allow us to show that I(/+oov)  % CIIvll,:,« So 
(2.10) holds. By interpolation, the result remains valid for every s > s0. 

3. ANALYSIS OF THE LINEAR EQUATION 
AND PROOF OF THEOREMS 1.1 AND 1.2 

Let us consider the operator (1.2), where we take the coefficients bo(x, y) and 
ao(x,y) in the space G(q'q)(ft) with 1 < qt < , q2 > 1. We choose t in (1.2) in 
such a way that there exist two integers  and  for which  ' 
-ff += rn- t. 
Let us observe that the operator P(x,y, D,Du) does hot involve the terlns 
l j m_t_" __ 
(x, y)Dx D with order I -  < m-; these terres have order I  +j  -- e, 



3416 GIUSEPPE DE DONNO AND ALESSANDRO OLIARO 

where et is giveu by the following expressiou: 

(3.1) 

(3.2) 

P(x,y, Dz, Du)v(x,y) = f(x,V), 

P as in (1.2); so it is uot restrictive to multiply the coeflïcients bo(x, y) and aU(x, y) 
by a flmction \(x, )  Çq'q)() with support in a neighborhood of the origin. 
Thus, we tan suppose that (x, ) and aq(z, ) are compactly supported. 
Now we fix a weight fllnction (, ç) of order (r, d 
), essentially subadditive with 
respect to ç; for every s  s0 and ç we consider the anisotropic Gevrey-Sobolev 
' ( x (-& )). 
space ,  « 
For an arbitrary real number q 2 1 we set: 
(i) r,q,r,comp(Q)s' := {f  r.q,rS'¢ (N X (--5, 5)) with compact support contained 
in Q}; 
(ii) N:'toc().,, := {f e '(Q) " for every  e Gq'q2)(), 1 < q < , q2 > 1, 
s, 
we have f  Nr,q,r,comp(Q)} ; 
(iii) s, 
Nr,q,r(Q ) := {f such that f is a restriction to  of a function belonging to 
,¢ (_,))}. 
,q«( x 
It follows from Theorem 2.1, Theorem 2.3 and Renmrk 2.4 that, for s  s0, 

(3.3) P(x,y, Dz,D)" ' NI;-((1R (-,)). 
..,,.( x (-,5,,5)) ----, x 

Moreover, the operator P(x, y, Dz, D) can be regarded as a continuous map 

P(x, y, D, D) ' -'' 
 B.,«,«o(fi) -- 

or also as a COlltinuou8 map 

P(z, y, Dz D v)  s, 

Now let. us consider the following operator: 

(3.4) 

P(x, y, D, Dy) := e'-W('D) p(x, y, Dz, D)e -w('D«). 

By Remark 2.1 and the previous considerations we have 

P(x, , Dz, Dy) = e-rP(Y'D*) )(X, y, D z, Dy)e rp(y'D*) , 

and, inoreover, 

(3.5) 

(x,,D,D)" H(N x (-5,5)) H-(N x (-5,5)). 

Now we want to analyze the conjugate operator /3(x, y, Dz, D); to this aire we 
start from the following proposition. 

The symbol .M(b) stands for the decimal part of b, and It = {h E [0, rn - t), h E 
N  (m - t - h) ¢ N}. We deal with the local solvability at the origin of the 
equation 
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1 Let us riz 
Proposition 3.1. Let O < p <_ 1 and O < r < 1; as usual we set q = . 
a function a(x, )  G q' 'q) 1 < q <  q > 1 and a weight function (U, ç) of 
order (r, p), essentmll sbaddtive wth espect fo . The 
e r¢(,D ) a( x, y )e- r¢(,D ) 
(3.6) = a(x,y) + Q_(q_)(x,y,D) + Q_2(q_)(x,y.D.), 
where the symbols q_j(q_)(.r, y,ç) of the operators Q_j(q_)(x, y, Dx). j = 1, 2. 
satisfy the following conditions: 
ç-(q-)p«o ); 
q_(q_)(x,y,ç) = rOç¢(y,ç)(Dxa)(x,y)  1,0 k X 
-(q)» " ). 
Proof. Setting Q(x, y, Dz) = e¢(v'D)a(x, y)e -¢(v'D) and using thc standard 
properties of the oscillatory integrals, we eily obtain that the symbol q«(x, y, ç) 
of the operator Q«(x, y, Dx) is given by 
(3.7) q.(x, y, ) = / eiXV«¢('ç+o)-¢('ç)à(, y) dq, 
where we write dq = (2) -1 dl and we d('note  usual by (,y) the Fourier 
transform of a(x, y) with respect to the x variablc. Applying the Taylor formula to 
e r¢(v'ç+v), we obtain 
eV¢(,ç+v)-r¢(,ç) 
(.s) 
N-1 
L (¢-¢('«) 0«¢('«))v  + 
=  + 0«¢(,ç) +   

where 
_ N fO  
(N- 1)! 

e -¢('¢+t)0 (e¢(v'¢+tn))e(v'¢+tn)-(v'¢)(1 - t) N-1 dt 

FtrOlll (3.8) and the standard properties of the Fourier transform it follows immedi- 
ately that 
q«(x, y,) = a(x, y) + "rO¢b(y,ç)( Dxa)(x, y) 
+. 
J 
n=2 
+ 
, () (,,1 + -() (,,1). 
= a(x,y) + q_(q_)(x,y,ç) + (q_2(q_) q-2(q-) 
rp 
Using the Leibniz rule, Definition 1.1 and the fact that  G SL0((-5,5 )  ), cf. 
Definition 2.1, we easily obtain that q-(q-r) G SI, 0 (  have onlv to 
-(q-)» ) j = 1, 2. 
prove that -() x, y, 
--2(q--r) k ' 
First we obtain by induction on n that for every j, k G Z+ there exists a constant 
Cjk such that 
(3.9) 
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Using the Leibniz rule and the estimate (3.9), it is easy to dedu«e that for every 
fixed N, _(x) x -2(q-r)p ]1). 
Now we «onsider q(q)(x, , ). Let us observe that, (, ) being essentially 
subadditive with respect to ç, we get e (v'ç+tn)-(v'ç)  e(X+[n[F'; moreover, 
due to }DD(V,)  Cdt(1 +}])pr- ( Ç S;), and to the inequality t+[ç+,n[ - 
1+In[ j 
, ]DvD e  + -- ¢(Y,))l < 
Indeed, 
j k j k j 
 c( +  + t,l) »- + c( + 
< 2c( + )»( + })( 
+ 
k 
-  + I/ + c( + 
 Ç( + )»-(1 + In[) »+. 
So we obtain, using Faà di Bruno's estimate, that 
O<hj+k 
' ((y,  + t,)- (y, ))I-. IDD((y,ç 
il +-.-+3a=3 
Using (3.9) with  + tq instead of , the previous estimate and the fact that 
x+lnl 
1 < we have 
(3.0) 
G Cj(1 + Iç) r0+)+rN+N+ ¢c(+lnlF'1 + I) r0+)+rN-N-. 
Reasoning as in the proof of Theorem 2.2, we find that, if a(x, g)  G q'q), there 
 Using this 
exists a constant M su«h that Dà(.l)[  Ce -M(+lniF', p = CG. 
fact. the estimate (3.10) and the Leibniz rule, we have 
DD D,() , , 
 «-(a-)x,,) S Q;( + 1)»(+)+» -- 
x/'(1 + ])»(+)+++[I%(+lnlF' -M(+lnl)' 
dq 
since pr < 1, taking N sufficiently large, depending on j and k, we obtain that 
0) "x , , d2) t x 
the symbol q_2(q_r)(X,y,ç) = q-2(q-r)k ,Y ç) + q-2(q-r)ç 'Y') is in the «lass 
Sa, o ( x 
Definition 3.1. Let us consider a function a(x, y, , ç) G Coe( x 2) and define 
(3.) »(,,) = ( + I1 =» + I,=)   + I[  + @. 
We say that a(x, y,, ç) Ç S(p,)( x 2), p G 1, if for every l, j, k. h G Z+ there 
exists a constant CIjkh sHch that 
(3.12) t j k h . 
. IDzDyDçDa(x,y,,)  CtjkhAp(,)(1 + ])pu--pa 
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Proposition 3.2. Let p, r, the function a(x, y) and the weight function .,(y, 4) be 
fixed as in Proposition 3.1. Then 
{T¢(y,Dx) l 3 --'¢(y,Dx 
a(x, y)DDe ) 
(3.13) ----a(,r,  J D)D«D J- 
) - j(Æ, )(¢)(, 
+ A+, ,x{_O_,__)} (x, y, D» D), 
where the symbol a . 
+, mx{--2(--),--(q--)} (X, y, ç. ,1) of the pseudo-diçerential op- 
emtor A +j, {_2(l_),_(q_)}(x, y, D, Dy) belongs to the class 
S+j,x{-(-),-(-)}( x ). 
Remark 3.1. The operator -jra(x g)( D)(y,  J- 
D)D«D in (3.13) bas the svm- 
.+,-0 
bol -jra(,g)(D)(g,ç)çt! j- in the elass (ç.1) -)( x ),  is easy to 
rp 
prove since   SLo. 
Proof. Using thc idcntities (2.2)-(2.4) and Proposition 3.1, we gct 
c(,D) a(x, ID  DJ e-(,D«) 
a x 
: ¢r¢(Y'D*)a(x, y)e-r¢(Y,D)¢r¢(Y,D*)D  
x y 
l j I 
 ( - j(¢)(, + (, 
Dæ)DæD  qj-hAJ) D)DD) 

l j l j-1 
= a.(«, y)Dfl) - jTa(«, y)(D)(, Dx)DxD 
+ A_+j, ma×{_2(l_r),_(q_r)} (X, y, D, D). 
In the expression of the symbol of the lt operator, let us analyze, for exanlple. 
the terre a(x,y)qJh(y.ç)ç h, for h 0, . ,fi 2.  get, for ,j ,  N+ and 
l' " k' h' 
 Ct,j,,h,   IDD IDF"ç j-h(B,ç)llçl-]çl - 
" k +kz=k 
5 Cl'j'k'h' (1 + [Ç])pr(j-h)-k (1 + [Ç[)l-kAp(Ç, 1]) h-h' 
 Ct,¢,,h, (ç, )+-al-)-'(1 + Içl) 
 Cz,,,, (ç, )+(1 + Içl) -(x-)-'-w. 
¢rp s.pr(j--h) 
In general, using the Leibniz rule, the estimates on ¢  '1.0 and qj_h  ,o , 
and the fact that the symbols of the operators q-k(q-)(x,y, D) are in the c]s 
l,0-k(q-r)P( x N) for k = 1, 2, we obtain: 
qj._hty, ç)çç h is in the class 
h=0 
S+'-a-( x N); 
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 (q-(q_r)(x,y,) + q-2(q-r)(x,y,))(r J - jr(Dy¢)(y,)r j-l) is in the 
÷j,--(q--r) ( X ]12). 
class S(p, 1) 
+, +,-(-)( 
Let us observe that S(p,1 ) -2(1-r)( X N 2) and S(p,1 ) X N 2) are both 
contained in S+j'max{--2(--r)'--(q--r)}( X N2); SO it foIlows immediately that 
(p,1) 
+j .max{--2(--r),--(q--r)} ( X R2). 
al  
+, max{-2(1-r),-(q-r)}( x, 9, (, ) Ç 
Now we want to analvze the behavior of the conjugate operator, defined by (3.4). 
We choose the weight function in the following way: 
where 0 < r < 1,  > 0 raid (() is a C  fimction such that 0 N (()  1, (() = 0 
for Içl   and ç(() = 1 for I(I OE . Observe that ¢(y,() is a weight mnction of 
order (r, ), essentially subadditive with respect to (. 
Proposition 3.3. Let us fix the operator P(x, y, D, D) as in (1.2) with d < m, 
and let us fix 0 < r < 1; we coose the weight function as in (3.14). Then the 
symbol of the conjugate operator is given by 
(x,y,(.,I)=p(x,y.,,1)-i(ç)[[ +pm,-(-)-.(x,y.ç.ç) 

where p(x, y. 4. 7) is the symbol of P and 

m,-(-)-.fi x R2), u> O. 
Observe that 

Pro@ First we observe that we can write the operator P(x, y, D, D) in the fol- 
lowing way: 
m l j 
(3.15) P(x,y,D,D) = Dy -bo(x,y)D +  atj(x,y)DDy, 
m--tl +jm--eo 
where e0 is given by (3.1) with t = 0. 
a and   in 
Now, applying Proposition 3.1 and Proposition 3.2 with p =  
(3.14), and using (2.4), we get 
(x, y, D, D) = eV('V)De -v('v) - eV('V)bo(x, 
+  e('D)aj( x , ,,D  DJe -('D) 
m-tl  +jm-eo 
m 
= Dy - mr(Dy¢)(y,D)D - + A,,.mx{_2(_),_(_)}(x,y,D,Dy) 
+  (,) + 
" +j 
m-tl _ 
= e(,y,,) - (¢)(, )2 - + 
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where 

The following theorem allows us to find a parametrix of the operator (1.2) when 
m - t = --, for a positive integer k* such that d(m - ) < k* 
2 < dru and the set 
Ik. := {(/,j) e Z  dj+ml = k*} consists ofjust one couple t ,3 )- Wetake 
b0(z, y) and aj(, 9) in C(). 
Theorenl .1 (Sobolev parametrix). Assume ha (1.5) ,n Theorem 1.1 and (1.3) 
hold. Then flere ezists a lin«ar map 
E  H» ( x (-&))  H[ ( x 

such that 

P(x, y, D,, D)E.u = 0(x, y)u + 
where 9(x.y)  C , O(.T.y) = 1 in a neighborhood of he origin and R is a 
regularizing map in he Sobolev anisotropic spaces. 
emoving hypothesis i) in Theorem 1.1, we shall prove that there exsts a 
1 where nov we take 
Gevrey-Sobolev parametrix of the operator (1.2) for t < , 
analytic coecient.s, or more generally coecients in the anisotropic Gevrey space 
Gql'q2)(), 1 < ql < m' q2 > 1 (cf. Proposition 3.1), with r > max{ ,1- t- et}. 
Theorem 3.2 (Gevrey-Sobolcv paramctrix). Let one 4 the conditions lai or lb] 
in Theorem 1. and (1.3) hold. Then there ezists a linear map 
.,  .,. (  (-,6)) 

such that 

P(x, y, D., Dv)Eu = \(x, y)u + Ru, 
where ,((x, g)  G(o')(f), \(x,y) = 1 in a neighborhood of the origin, and R is a 
regularizing map in the anisotropic Gevy-Sobolev spaces. 
The proof of Theorems 3.1 and 3.2 will be deduced froln Theorem 3.3, below, 
about the hypoellipticity of the following class of differential polynomials with C 
coefficients h(.,.) :   C: 
(3.16) p(x,g,ç,ç) = 
(l,j)I 
for Ç = (ç, ç) the dual variable of z = (z. ). Ve define the following sers for 
k E N+,O < k < dru: 
Ik = { (l,j)  N+ xE+ : dj + ml = k}, 
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1 
and fix k = k* such that d(m- 5) < < dru. We use the notation k- for ail 
k < k* and k + for all k > k*. We define I = I_ U Ia. U I+, with I_ = U I-, 
I+ = U I+. The swnbol a(,9,¢,) in C(N  x N) is such that 
k* 
 DI D  °  t , _ , 
(3.17) [D u «,'Ç)[ < Co(1 + A(Ç)) -(+°) m < d 
d 
wherc A(Ç) : [[  + [[ is the anisotropic norm; cf. the expression of thc anisotropic 
Sobolev spaces in Definition 2.2: H,)(), s OE 0. We recall that E :: {(z,Ç)  
 x z  {0}  ! TM - Rbdo d : 0} is thc anisotropic charteristic manifold of 
p(x,y,,l) in (3.16); letting A be a neighborhood of E, we denote by F the set 
l x A, and we state the following: 
Theorem 3.3. Assume that Ik. consists of just one couple (i*,j*), k* : dj* + ml*, 
such that: 

i) C3hl.j.(x,y) ¢ 0 fo-," ali(x,y) C 12, 

il) fo," ail (l,j) such that dj + ml > k* = dj* + mi*, 

(3.18) 

iii) ht.j.(x,y),hdO(X,y)J* d+t* <_ O, (z,Ç) G F, 

il ?) ïà0(3",]) ¢ O, fOl " ail (x,y) E . 
Then 
(3.19) IP(«,Y,¢,ç) k bA(Ç) i  x Nz, 
for asuitabl« «onstantb. Also. for alt (a.Z) e Z, (%0) e Z and for ail Iç CC a 
.we ha've, with suitable constants L,o,,o and B, that 
(3.20) 
  3 0 
 L,«,o,  +  > , 
k*--d(m--1)  = dru--k* 
with p -  ,  Obseçe that  < p, since we bave assumed 
* > (-). 
Remark 3.2. By formula (3.20) and by Mcarello and Rodino ([MR], Theorem 
3.3.6), we have that the operator P(z, D), associated to the symbol p(z, Ç) in (3.16), 
is C-microlocally hypoelliptic in F; i.e., F  WFPu = F  WFu, for all u G 
D (), where II'F u is the H5rmander wave front set. A nficrohypoelliptic operator 
is hypoelliptic too. If the coecients are analytic and (3.17) holds for Co = 
L+++°+lçO, we obtain G«-hvpoellipticity, « > 
 -- k*-d(m--1)" 
Before starting the proof of Theorems 3.1 and 3.2. we also need the following 
cls of symbols: 
Definition 3.2. Let a(x,y,,)  C(  ). We say that a(x,y,,)  
S'l(v')( x z) if for every l,j, k. h  Z+ there exists a constant. Ct- such that 
t j k h  , 
(3.m) ooo«o(x,v,,v)[  GA,)- - 
where £p(, 1) is given by (3.11). 
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Further on we shall also refer to the microlocal classes of symbols 
1,0 (F), where now F is a quasi-homogeneous cone.  leave to the reader the 
obvious definflions in this frame. 
In the following we suppose that tlw quai-homogeneous cone F is included in a 
region in which [[P  c0ç (in particular wc are intcrested in the case p = ); for 
these sets we have that (1 + ])P  kv( , ri), and so S/ï){F) can be identified with 
m+p:(p,1) 
1,0 

Definition 3.3. Let q(x,.tl,, q) E ,ç(»'l/(F). Let us sui»pose that m' _< m a.lld 
0 N  < p  1.  say that q(x.y..q) is of type (m.m'.p.p. 6) if thcre exist 
positive constants c, C Çljkh Slich that in F the following estimates hokl: 
(3.22) Iq(x, y,,,)l  c(,,) m , 
(3.23) 
[DDDç Dq(a. y, (. 0)[  Gjkh[q( x. ., , 1)[ Av(Ç, l])-p(+h)+5(+j) 
for v((. q) 2 C. 
Proposition 3.4. Let q(x, y, (. q) be of the type (m../, p. p. ). Then there exists 
a symbolq'(x.y..i) Ç p, (I'). i.e., 
D 
q' bei 9 the parametrix of q(z. , (, 0), i.e.. qq'  q'q  1, where  idicates the 
standard as9mptotic product. 

For the proof of this proposition, sec Hunt and Piriou [HP]. 

Proof of Theorem 3. I (Sobolev parametrix). Without loss of generality, we nmy as- 
sume po : (0. s/0) with 0 > 0. The quasi-homogelmous conic neighborhood F 
of p0 is then included in a region { > 0, ç2 < C2}. Since P satisfies ail the 
hypotheses of Theorem 3.3. we bave that the Svlnbol p(x, y,ç, 1) is of the type 
(   k*-d(m-) dru-k*) So using Proposition 3.4 we can find a linear map 
" '' d  d - " 
E  H  H " sueh that PE = O(x,y)p(D»D)+R, vhere O(x,y)  C(), 
P(-0) Ç COe(2) with support in a quasi-homogeneous conic neighborhood of 
(0, 0),   £ (  (-6,))   (  (-6,6))  vy t OE 0. s,, e , 
struct two operators E1 and R su«h that PE = (x, y)p(D, D) + R, where we 
can «hoose the fun«tion O(, '/) e C  quasi-homogeneous in (¢, '/) of order (, 1) 
and of degree 0 for large ([(I  + IV[), O((,V) = 0 in a quasi-homogeneous conic 
neighborhood of ( = 0 and ,  H (N x (-3,3))  H (N x (-3,)) for every 
t  0. We can suppose that supp(1 - O((, q)) C F0, where F0 is a suciently small 
neighborhood of ç = 0 such that 0 m -b0(z, 9)(d is quasi-elliptic in ç0- In the follow- 
ing we denote by m(x, , (, 0) the mfisotropic principal svmbol of P(x, , D, D), 
i.e., (z, , , 1) = V  - ho(x, )(d. By the results of Hunt and Piriou [HP] we 
hae, in F0, 
p(z, ,ç, v)#  (x, y, , o) 
=  
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 d 
S- ';( 3) 
where q_, C 0,0 and cm is given bv (3.1). In a standard wa) we can 
construct s_«  o,o such that 
(1 +q_(x,y,ç,W))#(1 + s-(x,y,ç,,I)) 1. 
Let us consider now the svmbol 
«0 = Ll(x, Y, ç,'l)#(1 + s_,(x,y,,q))#O(x,y)(1 - 
and let E0 1)e the pseudo-differential operator of synlbol e0. Bv construction we 
obtain PEo = 0(a', y)(1 - o(D, Dr)) + Ro, with R0 regularizing on the anisotropic 
Sol)olev spaces H (R x (-& )). Taking E = E + E0, we have that 
PE = PE + PEo = O(z, y) + R, 
where R = R + R0 is regularizing on H (R x (-6, )). 
m 
Proof of Tbeorem 3.2 (Gevrey-Sobolev pammetrix). Let us suppose that one of the 
global conditions [a] or [bi holds. When [ai is satisfied we fix the weight function  
in (3.14)-if [bi holds we «hooc (y.)= (1 +  sir,, ç )(ç)çl{  ((y,ç)= (1- 
 sig,,  )ç()[([ ). Bv Pr()position 3.3, thc svmbol (x, y, , l) of the «onjugate 
Ol)crator (x, y, D, Du) defined by (3.4) satisfies all the hypotheses of Theorem 
3.3 with j* = m - 1, l* = r. So (.,', y, ç, 'l) is of type (,n, m - (1 - r),p, r. 1 - r). 
As in the first part of the proof of Theorem 3.1, by Proposition 3.4 we can find a 
linem" map 
Ê" H (R x (-& 5)) + H+-(1-ç)(R x (-&5)) 

such that 

Ê= \(a.,9).(D.,Du)+ , 
where ¥(;r, .) is arbitrarily fixed in C(t), (ç, q) is arbitrarilv fixed in C( z) 
vith support in a quasi-homogeneous conic neighborhood of (0, 0), and  is a 
regularizing operator in the anisotropic Sobolev spaces H. So we can find Ê and 
1 sR'h that ?Ê = k(x,y)(D,Dy) + 1, where we can choose the C oe function 
(, q) with the properties of the function 0(, y) in the previous proof ((, q) 
qui-homogeneous of degree 0 out of the origin, (, q) = 0 in a neighborhood 
of  = 0 and supp(1 -(,q)) C F0, where çm(«,y,{,l) is quasi-elliptic in F0)" 
moreover,   H (N x (-&6)) + H (N x (-&6)) for everv t 2 0. 
Reasoning as in the preceding proof and using Proposition 3.3 and the results of 
Hunt and Piriou [HP], we have, in F0, 
(z, u,ç, )# (, u, ç, v) 
= ((x, , ç, 0) + q,,,-m.{.l-} (x, , . ))#p£1 (, , ç. ) 
= 1 + q- i.{=,l-} (Z, y, , 
ç-mi{,-};({,) In a standard 
where the symbol q_ mi{,-} is in the class 0.0 
S- i{,-};(  
 such that 
way we can construct s_ rein{cm,l-r} Ç 0.0 
( + q- i..l-}(, y,L,))#( + s-mi.{=.l-}(*, > ç, V)) . 
Now let us consider the pseudo-differential operator Ê2 of the symbol 
ê = p.,,,--*(, , ç, )#( + _...{,l_}(a,>,v))#x(Æ,)( - (ç, )). 
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By construction we obtain/Ê2 = ,x (x, y)( 1 - n(D«, Dy)) + 2 with/2 regularizing 
on the anisotropic Sobolev spaces H_( × (-&3)). Let Ê = Ê1 + Ê2. Then 
PE = PE1 + PE2 = ((x, g) + R, R regularizing. 
After conjugation we obtain PE = e--¢(v'D*)\(x,g)e -¢(v'D*) + R. Taking a 
function \o(x,y)  G(oq'q=)(Ç) such that ,\o(x,y) = 1 fol (.r.g)  supp(x) and 
replacing E by E,, where Eu := E(X(x, g)u), we have 

P(x, y, D, Dy)E,u = e--¢'(Y'D).\o(X, g)e -¢'(y'D«) \(x, g)u + u. 
= \(x, y)u -- e-r¢(Y'D)Ï.3er¢(Y'D)u + -u. 

The operator i3 = (1 - \0(x, y))e(Y'D*))(x, y)e -(y'D«) is again regularizing on 
the anisotropic Sobolev spaces H (IR × (-. )) in view of Proposition 3.1. [] 
m 

Remark 3.3. Theorem 3.3, Remark 3.2, and Theoreln 3.1, COlnbined with fixed 
point argmnents as iii Gramchev and Rodino [GR, Section 4], lcad to the proofs of 
Theorem 1.1 and Theorem 1.2. 

Proof of Theorem 3.3. We linfit ourselves for silnplicity to proving the estimate 
(3.20) for c +/3 + 7 + 0 = 1. The case c + ]3 + 7 + 0 > 1 does hOt involve actual 
complications; cf. Wakabayashi ([W],Theorem 2.6) or Kajitani and Wakabayashi 
([KW], Theorem 1.9) for the analytic frame. First we estilnate the lmlnerator of 
(3.20), and then we give some lemmas to estilnate tlle denolnillator. 
If a = 1, we get 

 [ (l,j)Z I 

and for/3 -= 1, 

and for 0 = 1, 

]DrlP(z'Ç)I/(Ç)P < L4(( Z I 71-x(+lz/[-l)'x(Ç) ° + A(Ç)-0-P)) , 
(td)  I 

with suitable constants L3, L4. 
Therefore, we observe that  - (1 - p) _>  - (1 - p) since d < rn, and 
rn--(1-p) =-( >-(sincep+(= 1. Toprove (3.20),it willthen be 
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sufficient to show the boundedness, for ]Ç] > B, of the functions 

Q«(ç) = 
First we introduce three regions: 

(3.24) 

n,  c KI d  Iwl r  c KI d, 
R2. Iwl r  cKI d, 
R- Iw, r  c KI d, 

for suitable costa.nts c,C satisf.ving c < 7 nllll(x,) f ]N ho(x.y)], and C > 
21nax(,v)e ].Rh«o(x,y)]. VVe understand the neighborhood A to be the region 
l- 
The following inequalities then hold: 
{ Clvl - ç  , () 
(3.25) (Ç)-6  Ivl -a, çe R2, (II) 
[-, Ç  R3, (III) 

and 

{ C1 IT/I p, Ç E Ri, 
(ç)p  c21wl p, ç E R2, 
note that (II) and (III) in (3.25) hold for ail Ç G N 2, but for out aire we mav limit 
ourselves to consider them respectively in R2 and in R3. By abuse of notation, in 
the following we shall atso denote by RI, R.2, R the sets f2 x RI, f2 x R2, f2 x R3; 
recall that F = f2 x A. 

Lemlna 3.1. Let p(z,Ç) be the function (3.16), s.uch that (i),(ii),(iii) in (3.18) 
hold. Then there are positive constants h' < 1 and B, such that 

(3.26) 

Ip(z,ç)l :>/<, Ight..(x,u)l I,l'l,[ t', (z,ç) E R, Içl > B. 

Pro@ We have 
Ip(,ç)l 2 
(3.27) 
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where 
(3.28) 

By removing the terres arising from the real part of p(z, Ç), we can write 

(l,3)El_ 

5 
IP(z,Ç)i2  hl*J*('r'Y)2y]2j*Ç21* -[- Z 

k hlj (z)r]3ç I + 

 hj (z)rF¢ t - hdo(Z)¢ d + «(z, 
(l,3)el+ 

(3.29) J2(z,Ç) = 2hl.j.(x,y) Z 
(l,j)l_ 

(3.30) J3(z,Ç) = 2ht.j.(.,y) Z 

 ht(x, y) lJ'+Jç '+ 

(3.31) 

da(z, ç) = -2 

(3.32) 

Js(z,Ç) = 2La(z,Ç)bt.j.(x,y). 

The function (3.28) is nonnegative: (3.30) and (3.31) are also nonnegative by hy- 
potheses (ii), (iii). Let us fix attention on J2(z, Ç), defined l,y (3.29). We have, for 
ail e > O, 

(c]ll.j. (:T, y))2/72j*Ç2/* q_ j2(z" Ç)  (1 - £)(c]ll.jo (x, /))2 112j'2l" , 
in/, ICI > B. In fact, by (3.24) in R and hypot.hesis (i) in (3.18), for ail e > 0 

we get, for B sufficiently large, 

(.ht..(x,))22.2t" <_ «o.st 

]2j ° 
(t,)e_ 

Weremarkthat k* = dj* + nl* > k- = dj + 
Concerning (3.32), since  < , we bave 

(hl-j* (x, y))2 r/2J* ç2* + J5(z, Ç) >_ (1 - e)(.hl.j. (x, y))2 r/2j-ç2- . 

Then 
Ip(z,ç)l E/t-i lh.. (.r, y) l IlJ'll t', 
fol" a suitable positive constant Içl. 

Lemma 3.2. Let p(z,Ç) be the function (3.16). Then there are positive constants 
K2 < 1 and B, such that 

(3.33) Ip(z,ç)l OE K2l]l TM, (z,Ç) e t2, Içl > B. 
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Pro@ We write Ip(z,Ç)l 2 as in (3.27); by removing the terres arising from the 
imaginary part of p(z, Ç), we get 

(3.34) 

where 

(3.35) 

Ip(,ç)l 2 2 (,!  - .ho(x,v)U) 2 ÷ wl(,ç) ÷ U(z,ç) ÷ W3(z,ç), 

2 
,r h o (z, ) n t + .«(z, ç)) , 

(3.36) 

(I,j)EI 
- 2ho(x,)  .(x,y)( +e. 
(l,j)eI 

(3.37) H3(z, Ç)= 2q'cr(z, Ç)- 2.ho(x.y)cr(z,Ç). 
Observe first that, for A > 0 suciently small, 
( - ho(x,u)U)  > . 
In fact, 
(, -  ho(X, ))  .f - 2 ho(z, ) n  , 
and using (3.24) in R2, we have 
q - 2Nhao(x,)ç( a 2 1- ÇNhao(x, 9) > , 
since C > 2 max(,ea IN hao(z, 9)l- 
Equation (a.aa/is nonnegative.  denote (a.a6) by (z, ç) - (z, ç) and 
(a.aT) by a(z, Ç) - 4(z, Ç). Then 
Arguing on  - , Ta - 4 in the saine way as we did in Lemma a.1, we tan 
show that for ail e > 0, 
,2 + T(,ç) - T(,ç)  (-), (z,ç)  n, Içl > U, 

and 

Thus 

,xr 2" + Ta(z,ç) - T4(z,ç) > (- )rf ", 

>(,ç)l /% I1  , 
1 
where K2 = (A - e). 

[] 

Lemma 3.3. Letp(z,Ç) be the function (3.16), such that (iv)in (3.18) holds. Then 
there are positive constants Iça < 1 ad B such that 
(3.38) Ip(z,Ç)l _> Iç3 I(I a , (z,Ç)  R3, IÇI > B. 
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Proof. Again we apply (3.34), (3.35), (3.36), (3.37) to Ip(z,ç)l 2. Observe that in 
 min(=,u)En phdo(x, Y)I, we obtain, for a suitable 
D3, arguing as above, since c <  
constant it > O, 

(o" -lhao(-r,.V)(;a) 2 > I' 

About the terlns in (3.35), (3.36) a.nd (3.37), the relnarks we ma(te ill Lemma 3.2 
hold on replacing A q2,, with p s c2d. Then we bave 

I»(z,ç)l  311 , (=,ç) e/¢3, ICI > B, 

1 
where K3 = (p - ). [] 

We first considcr QI(Ç) separately in the regions R, R2, Ha, to prove bomd- 
edness. In H1 bv (3.25), (3.26) we get easily, writing as before k = d3 + ml, 

Q,(ç) _< co,,st ,,__ +1 , ICI > B, 

where rn - â > 0 by definition of I and I. In the region R2 we have Ip(z, Ç)I -> 
I,1TM > M  . h,/3, b»- using (3.25) and (3.38) for a constant « > 0 which we mav 
take as small as we want by fixing/3 suflïciently large, we have 

çVe have therefore proved that QI (Ç) is bounded. Arguing in the saine wav on 
Q2(Ç), Q3(Ç) and Q4(Ç), we prove their boul,dedlmsS in . 

Re'mark 3.4. By formulas (3.26), (3.33), (3.38), we ol,tain that Ip(,ç)l  lçl -, 
a > 0, IÇI > /3, since we are considering the case when dj + ml < dru. If we refer 
to the anisotropic weight fimction A(ç) = I{l + Irai ,- (1{I d + [r/l'), we find that 

(3.39) Ip(z,ç)l  &m(ç)-, ICI > B, 

for a suitable positive constant b. 
Now Lemma 3.1, Lemma 3.2, Lemma 3.3 and the estimate (3.39) complete the 
proof. [] 

Re'mark 3.5. It is possible to propose a geometric invariant generalization of Theo- 
rems 1.1 and 1.2, to pseudo-differential operators with involutive characteristics of 
nmltiplicity m >_ 4, in more than two space variables, by arguing nficroloca.lly, using 
classical Fourier integral operators and Sp, arguments. This will be the subject of 
another paper. 
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ACCELERATING THE CONVERGENCE 
OF THE METHOD OF ALTERNATING PROJECTIONS 

HEINZ H. BAUSCHKE, FRANK DEUTSCH, HEIN HUNDAL, AND SUNG-HO PARK 

ABSTRACT. The powerful von Neumann-Halperin method of alternating pro- 
jections (MAP) is an algorithm for determining the best approximation to any 
given point in a Hilbert space from the intersection of a finite number of sub- 
spaces. It achieves this by reducing the problem to an iterative scheme which 
involves only computing best approximations from the individual subspaces 
which make up the intersection. The main practical drawback of this algo- 
rithm, at least for some applications, is that the method is slowly convergent. 
In this paper, we consider a general class of iterative methods which includes 
the MAP as a special case. For such methods, we study an "accelerated" ver- 
sion of this algorithm that was considered earlier by Gubin, Polyak, and Raik 
(1967) and by Gearhart and Koshy (1989). We show that the accelerated al- 
gorithm converges faster than the MAP in the case of two subspaces, but is, in 
general, not faster than the MAP for more than two subspaces! However, for a 
"symmetric" version of the MAP, the accelerated algorithm always converges 
faster for any number of subspaces. (ï)ur proof seems to require the use of the 
Spectral Theorem for selfadjoint mappings. 

1. INTRODUCTION 
Let X be a (real) Hilbert space, let/111,/lI2 .... ,/lIk be closed (linear) subspaces 
of X with M = Nî/iii, and for any closed subspace N of X, let PN denote the 
orthogonal projection onto N. The von Neunmnn-Halperin method of alternating 
projections, or MAP for short, is an iterative algorithm for determining the best 
approximation PMx to x from/lI. It does this by computing the iterates x0 := x and 
Xn - ( PlIk PIIk_, " " Phil )Zn-1 = ( PII PlI_l " " Phi1 ) riz. That is, the iterates (xn) 
are obtained by cyclically computing the best approxilnations onto the individual 
subspaces/iii (i = 1, 2,..., k). The lnethod is thus lnOSt effective when it is "easy" 
to compute the best approxinmtions froln the individual subspaces/lI. The main 
theorem governing the MAP is the following. 
Theorem (von Neumann [18] for k = 2, Halperin [15] for k _> 2). Let lll,llI2 .... , 
llIk be closed subspaces in the Hilbert space X and let lil := Nî llli. Then 
lira II(PM«PM_,...PM,)'x--PMxlI=O forall xeX. 
In case k = 2, this result was rediscovered in at least six other papers (see, e.g., 
the survey [5]). 
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Also, as was noted in [5], there are at least ten different areas of mathematics 
in which the MAP has proved useful. However, the main practical drawback of the 
MAP appears to be that it is often slowly convergent. Indeed, if M1 + M2 is hot 
closed, then Franchetti and Light [11] and Bauschke. Borwein. and Lewis [2] have 
given examples showing that the convergence of (PMPh)'x to PMIcMx can be 
arbitrarilv slov! 
Both Gubin, Polyak. and 1Raik [14] and Gearhart and Koshy [13] have con- 
sidered a geometrically appealing method to accelerate the MAP. but in neither 
of these two papers was it proved that the acceleration scheme considered was 
actually faster than the MAP. In this paper, we will prove that this accelera- 
tion scheme is indeed faster than the MAP in the case of two subspaces (i.e., 
k = 2) (Theorem 3.23). But, perhaps surprisingly, we show that the acceleration 
scheme ma.v actually be slower than the MAP v«hen k 2 3 (,Example 3.24)! In 
contrast to this, we show that a "symmetric" version of the MAP (i.e., x0 = x 
and xn = (PM, P"" PMPM_,''" PM,)nx for n = 1.2 .... ) has an accelerated 
version which is faster for anv k >_ 2 (Corollary 3.21). 
We should also mention that Dvcr [10] and Hanke and Niethmmner [16] have 
considcred nlethods of accelerating flic "Kaczmarz mcthod'" of solving linear equa- 
tions. (Recall that Kaczmarz's method may be regarded as the special case of the 
MAP in the case when ¥ is finite-dimensional and each Mi is a hyperplane.) 

2. THE METHOD OF ITERATED PROJECTIONS 
To provide motivation for the acceleration results to be established later, in this 
section we give a fairly general convergence result which contains the von Neumann- 
Halperin result as a special case. In the next section, we will consider methods to 
accelerate this general algorithm. 
Unless otherwise stated, the standing assumptions are as follows. Let X be a 
(real) Hilbert space, ,'I1, M2,..., Mk be closed subspaces. M " = Ç]î 3Ii, and let 
P, = PM, denote the orthogonal projection onto Mi (i : 1.2 ..... k). 
Now let 
T := Pk P-I'"P1 
denote the composition of the k projections Pi taken in increasing order. The 
well-known von Neunmnn-Halperin Theorem states that 
lira IITx - P^xll = 0 
for each x G X (see, more generally, Theorem 2.5 below). Also, it can be shown 
that 
lim I1(I"'I")'a: -- Paxll = 0 
for each x  X (see Theorem 2.6 below). More generally, suppose T is any bounded 
linear mapping from X into itself such that 
lim IITx- PoexTll = 0 for each 

(2.0.1) 
where 

FixT := {x  X I Tx = x} 
is the fixed point set for T. 
We will be interested in determining methods to accelerate the convergence of 
the sequence (T'x) to PFixTX. Before we consider such methods it will provide 
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useful motivation to first give some general conditions on the mapping T that will 
guarantee that (2.0.1) holds. 
The mapping T is called nonexpansive if IITll < 1. We first recall that the fixed 
point sers of T and T* are the saine if T is nonexpansive (see lIl.iesz and Sz.-Nagy 
[19] or lIl.iesz and Sz.-Nagy [20, p. 408]). 
Lemma 2.1. Let T be a nonexpansive linear operator on X. Then 
(2.1.1) FixT = FixT*. 
In fact, Tx = z if and only if (T.,x} = Ilzll oe if and only if (z,T*z} = Ilmll oe if 
and only if T*z = m. 
Out next observation is a characterization of those linear operators T on X that 
satisfy (2.0.1). We will use the following notation. If A is any linear operator on 
X, we denote the range and null space of A by T/(A) and Af(A) respectively. We 
will also use the well-known fact that Af(A*) ± = T/(A) (see [3, 1R.emarks following 
Theorem 2.19 on pp. 35-36]). 
Theorem 2.2. Let T be a bounded linear operator on X. and let BI be a closed 
linear sub.pace of X. Consider the following statements: 
(1) limn IITnz - Pzmll = 0 for each :c E X: 
(2) BI = Fix T and T'z -- 0 for each 
(3) BI = FixT and T is "asymptotically regular", i.e., T'z - T'+tz -- 0 for 
each z  X. 
Then (1) ¢== (2) ===> (3). If, in addition, T is nonexpansive, then all three state- 
ments are equivalent. 
Pro@ Suppose (1) holds. If z /lI, then T'z  PMz = z. So by the continuity 
of T, 
Tz = T(lim T'a:) = lira T(T'a:) 
implies that a:  FixT, i.e., ]lI C FixT. 
Conversely, let y  FixT. Then Ty =  and an easy induction shows that 
 = T'y for each n. Thus y = T'  Py which implies  = P E ]lI. That is, 
Fix T C ]lI. Hence ]lI = Fix T. 
If x /lI ±, then 
T"a: = T'(PMJ_a:) -- P(P±a:) = 0. 
This proves (2). 
Now assume (2) holds and let a:  X. Then 
Thus (1) holds, and this establishes the equivalence of (1) and (2). 
Now suppose that (2) holds and a"  X. By the equivalence of (1) and (2), 
we have that T'x -- P^x md so T'x - T'+x -- Px - P^lx = 0. Thus T is 
asymptotically regular, and hence (3) holds. 
This proves the first statement of the theorem. To complete the proof, suppose 
(3) holds and let T be nonexpansive. Then FixT* = FixT = BI by Lemma 2.1. 
Then for any x  X, we have that T'(x-Tx) = T'x-T'+x -- O. Hence T'y -- 0 
for every y E 74(I- T) which implies, since IITll _< 1 by nonexpansiveness, that 
T'y -- 0 for every 
y G 74(I- T) = Af(I- T*) ± = (FixT*) ± = BI -c. 
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Thus, for any x E X, 
Tx = T'(PMx + Pt.x) = T(Ptx) + T'(Pt.x) = Ptx + T(Pt-x) --' Ptx, 
and this proves that (1) holds. [] 
Remark. Statement (3) does hot imply statement (1) in general. To see this, let X 
denote the Euclidean plane and let e = (1,0) and e = (0, 1) denote the canonical 
orthonornml basis vectors in X. Defining T" X --, X by Tx = [x(1) + x(2)]e, it 
is easy to verify that T'x = Tx for every  E N and every x  X, so that T is 
asymptotically regular, M := FixT = spane, T'(e + e) = 2e for every n, but 
Pt(el + e) = el # 2e = T'(ex + e) for every n. Thus, T'x - PM(x) when 
3"--el ÷2. 
Corollary 2.3. Let T be nonexpansive on X and I --- FixT. Then 
lira I{T'x- PMx[{ = 0 for all x  X 
if and only if T is asymptotically regular. 
Lemma 2.4. Let 1l[1,/L2,..., ]l[k be closed subspaces of the Hilbert space X. let 
,[ := Ç]î/li and let T := Pht P^tk_l "'" Phh. Then T is nonexpansive and 
FixT = FixT* = Fix (TT*) = Fix (T'T) = M. 
Pro@ For simplicity, lct Pi = Phti. Since T is the product of nonexpansive opera- 
tors, T is nonexpansive. If x G M, then x G M, for each i so that Pix = x for each 
i and hence Tx = x. That is, M C Fix T. Conversely, if z E Fix T, then Tz = z. 
Thus, PkPk-''" Pz = z. W'e bave Piz = z if and only if {{Pz{{ = I{z{{ (using the 
fact that {{z{I 2 = {]Piz{{ 2 + {Iz- Piz{{2). If z ¢ M, let i be the smallest index such that 
z  /I. Then Piz # z; so {{Piz{{ < {{z{{ and z = Pk... PiPi-1 "'" Pz = Pk...Piz 
implies that {{z{{ = {{P--. Piz{[ < [{Piz{{ < {{z{{, whi«h is absurd. Thus, z  /I. 
This proves that M = Fix T. By Lemma 2.1, M = Fix T*. 
Since T* = PP2"'" P, we see that TT* = PPk- "" PP2"" Pk and T*T = 
PP2"' PPk-x"" P, and the saine proof as above shows that FixTT* = M = 
Fix T* T. [] 

A usefnl sufficient condition that guarantees that (2.0.1) holds is essentially con- 
tained in Halperin [15]. It also is explicit in Smarzewski [21] and can be stated as 
follows. (We include a brief proof since, as far as we know, the paper [21] bas hot 
been published.) Recall that T : X -- X is called nonnegative if (Tx, x) > 0 for 
all x G X. 

Theorem 2.5. Let TI,T2,...,Tk be selfadjoint, nonnegative, and nonexpansive 
bounded linear operators on the Hilbert space X. Let T := T1T2... Ta and Iii = 
Fix T. Then Fix T = Ç]î Fix Ti and 
(2.5.1) lira IIT'x - PMx[[ = 0 for «v«ry x  X. 
n 
Pro@ Since T is nonexpansive, Corollary 2.3 implies that it suffices to show that T 
is asymptotically regular. Toward this end, note that for each i, I-Ti is nonnegative 
(and selfadjoint) since 
((I- T)w,x) = (x - Tx.,x) = Il:r]] 2 - (Tx.,x) >_ I{x]{ 2 -{{T{{ {{xl{ 2 _> 0. 
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It follows from a result of Riesz (see [4, Theorem 4.6.4, p. 163]) that T/(I - T) is 
also nonnegative. Hcnce, 
Ilxll 2 = IIx- Tix -t- Tx[I  = I1- T,II  + ee<- Tx, Tx> + I111  
Thus, for each x  X, 
(oe.5.oe) I1- TII   I111  -IIT,II  for each i. 
By repeated application of (2.5.2), we dcduce that 
 I111  -I111  + I111  -I1-111  +" + I1 11  -I111  
 II - Txll 2 + IIT - T-ITXll 2 +-" + IIT... T« - Txll 2 
[ 1  ] 
=  I - r  + lr - r-r  +... + .lr... r - rl  
1 (X -- TkZ + %Z -- %_lTk Z +"" + T2..-Tkx -- Tz)[[ 2 
(by convexity of -]]2) 
1 
= 11 - Tll 2. 
That is, 
(2.5.3) II- TII 2  k(llll 2 -IITxll 2) fo every   X. 
Since T is nonexpansive, we see that the sequence (IIT'II).I is nonincreasing 
for every x  X and so it must converge: TI  p  0. Now apply (2.5.3) with 
x replaced by Tn to obtain that 
IIT - T+III2  (llTxll 2 -[/T+II[2 )  0    oe. 
This proves that T is ymptotically regular. 
Lemma 2.4 and Theorem 2.5 immediately imply the following t.wo results. The 
first is the "von Neumann-Halperin theorem" stated in the Introduction, while the 
oecond shows that a symmetric version of the MAP also converges. 
Theorem 2.6. Let I1, I2,- .., I be closed subspaces of the Hilbe space X, and 
let l =  M. Then, for each x  X, 
(2.6.1) lira II(Pl«Pz_,""" Pl,) - Plll = 0. 
Theorem 2.7. Let I1, I2,---, I be closed subspaces of the Hilbe space X. and 
let M = î M. Then, for each   X, 
(2.7.1) limII(PIPI'"PIPI_, ""Pl, FX- P*II =0. 
Using Theorems 2.6 and 2.7, we see that two important examples of operators 
T which satis (2.0.1) e T = Q and T = Q'Q, where Q := PMPI_ "'" Phil. 
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3. ACCELERATION METHODS 

Throughout this section, unless explicitly stated otherwise, we assume that T is 
a nonexpansive linear operator on X and M := Fix T. Hence, Fix T* = M also (b.v 
Lemma 2.1). Moreover, AI will always denote a closed linear subspace of X and 
Pi = PM,. 
In this section, we develop out main results eoncerned with aceelerating the 
nlethod given by (2.0.1). That is, if T is an operator such that (2.0.1) holds (or 
equivalently, that T is asymptotically regular), how ean we modify the iterates 
suggested by this algorithm so as to converge faster to Paix? 

Definition 3.1. The accelerated mapping AT of T is defined Oll X by 

(3.1.1) 
wh(,re 

AF(x) := txTx + (1 - tx)x, 

{ (.,.-:r) 
(3.1.2) t« = t.,T := IIx-Txll2 if Tx # x, 
1 if Tx = x. 

\'«' will ('(msi(lcr two clmsses of iterative algorithlns to compute PM(x) for any 
given .r  X. Thcy are descril»e(l as follows. The standard or "unaccelerated" 

algorithn: .fo = x and 
(3.1.3) .r = T(X=_l)= T"(«) 
and its "acc(']erat('d'" countcrpart: Xo = x, 

( = 1,2 .... ), 
Xl = T(xo), and 

(3.1.4) x, = AT(X,-I) = A-I(Tx) (n = 1,2 .... ). 
In particular, we will give a detailed analysis of these algorithms when T = PkP-I 
---P and when T = (PP_I.,. P1)*(PP-"" P1)- This acceleration scheme 
was suggestcd by Gubin et al [14] and Gearhart and Koshy [13] in the particular 
case when T is a product of projections. The inotivation for using the mapping AT 
is that AT(X) is that point on the line through the points x and Tx which is closest 
to PMx (see Theorem 3.7 below). 
A remark is in order as to why, in the aecelerated algorithm, we first apply T 
to x0 rather than first applying Aj . That is, why didn't we define the aeeelerated 
algorithm by xn = Aï(x0) for n _> 0 rather than x+ = Aï(Tx0) for  >_ 07 The 
simple answer is that, besides being the one suggested in/14[ aHd [13], the one we 
defined performs better. Indeed, it is hot hard to see that if T is the produet of 
two orthogonal projections onto two 1-dimensional (nonorthogonal) subspaees in 
the Euclidean plane, then the accelerated algorithm converges in two steps, that is, 
AT(Tx) = PMx for any starting point, x. However, for any choice of x which is hot 
in the range of T, none of the terres of the sequence (A(x)) is equal to Paix. That 
is, the sequence x = A(x) does hot converge to PMX in a finite number of steps. 
Definition 3.2. The classical von Neumann-Halperin method of alternating 
projections, or MAP for short, corresponds to (3.1.3) in the case when T = 
Pk P-I ""Pl. 
The accelerated method of alternating projections, or the accelerated 
MAP for short, is the algorithm (3.1.4) in the case when T = PP_... Pl. 
The symmetric method of alternating projections, or symmetric MAP 
for short, is just (3.1.3) iii the case when T = (PkPk-I"" P1)*(PkPk-I'"P). 
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The accelerated symmetric method of alternating projections, or ac- 
celerated symmetric MAP for short, is the algorithm (3.1.4) in the case when 
T = (PkPk-i'" "Pi)*(PkPk-i'" "P1). 
Lemma a.a. Let x  X. Then 
(1) rAT(x) + (1 -- t)x -- x  M ± ç (AT(x)) ± for every t  N. 
(2) Tx-x M±n(Ar(x)) ±. 
(3) AT(x) -- x  M'L n(AT(x)) z. 
(4) T(M -L) C M ± and AT(M ±) C M "L. 
(5) AT(X) - Tx  M "L n(AT(x)) "L. 
(6) Tx - P^x  BI "L. 
Pro@ (1) Since rAT(x) + (1 -- t)x -- x = ttz(Tx - x), it sufiïces to verify (2). 
(2) If Tx = x, then (2) is trivial. Thus we may assume that Tx  x. Let y  M. 
Then since Ty = y = T'y, we have 
(Tx - «, y} = (x, T*y} - (x, y} = (x, y} -(x,y} =0. 
Thus Tx - x  M "L. Also, 
(Tx- x, AT(x)) = (Tx- x,t«Tx + (1 - t)x) = t«(Tx- x, Tx- x) + (Tx- 
_ (x, Tx- x) 
- [ix - Txl] 2 [ITx - x][ 2 + (Tx - x, x) = 0: 
so Tx - x  (AT(x)) "L. 
(3) Take t= 1 in (1). 
(4) This follows from (2) and (3). 
(5) Since AT(X) -- Tx = (te - 1)(Tx - x), the result follows from (2). 
(6) Using part (2), we get 
Tx - P^x = (Tx - x) + (x - P^x)  I "L -t- M "L = 1ll "L. 

(3.5.1) 
In particular. 
(3.5.2) 
and 

(3.5.3) 

I[AT(x)- yl[ 2 = [Ix -- Yll 2 --I,x- AT(x)ll 2. 

FixAT = {x  x I IIAr(z)ll = Ilxll} 

ifxM, 
(x,x--Tx) 2 
IIz_Tzll 2 if X  M. 

Lemma 3.4. For every x  X and n  N := {1.2 ..... }, 
(1) PM(AT(x))= P^,rx; 
(2) P^(T:c)= P:c; 
(3) P^(A-I(Tx))= Px: 
(4) P^(T'x) -- PMx. 
Pro@ ç use the well-known fact that PM(M ) = {0}. Since Tx - x  5I  and 
Af(z) - z e M  by Lemma a.a, it follows that 0 = PM(Tx - x) = PM(Tx) - Px 
and 0 = PM(Ar(x) - x) = P(Ar(x)) - Px. Hence (1) and (2) follow. 
(3) and (4) follow by a repeated application of (1) and (2). 
Lemma 3.5. For each x  X and y  M, 
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Proof. Using Lemma 3.3, we deduce that 
IIx - .112 -- II(x - AT(X)) + (AT(x) - v)ll 2 = IIx - A:(x)ll u + IIA(x) - vll; 
so (3.5.1) holds. Take y = 0 in (3.5.1) to obtain (3.5.2). Finally, take y : 0 in 
__ (x,x-Tx) 2 
(3.5.1) and note that IIx - AT(x)]I 2 -- iix_Tl] 2 if x  Iii and ]Ix- AT(X)I] 2 = 0 if 
x E Iii. This yields (3.5.3). [] 
Lemma 3.6. The following statements are equivalent: 
(1) Tx e M; 
(2) Tx = P^lx; 
(3) Tnx  Iii for every n >_ l. 
Proof. (1) : (2). If Tx  Iii, thon Tx = PM(Tx) = PMx using Lemma 3.4(2). 
(2) == (3). IfTx = PMx, thcn Tx  Iii. Thus, (3) holds when n = l. We 
procecd by induction. If Tx  M for some n, OE l, then since/il = Fix T. we have 
that 
T+.r = T(T'x) = Tx  Iii. 
This completes the induction. 
(3) == (1). Take n-- l. [] 

The affine hull of a noneml)ty set S, denoted by aff(S), is the intersection of 
the collection of all affine sets which contain S. (Recall that an affine set is anv 
translation of a subspace.) Equivalently, aff(S) : {x + (1 - v)y I x, y  S, v ¢ }. 
Theorem 3.7. For each x  X and y ¢ 1I, we bave 
(3.7.1) IIAT(oe)--yll  = IItTx÷(1-t)x-yll2-(t-t)2lITx-xll  for each t E , 

(3.7.2) I[AT(x) -- Yll = rein IItTx ÷ (1 - t)x- yll, 
and the minimum is attained precisely when ezther t = t if x  M or af every 
t  1 if x  ]ll. Moreover, 
(3.7.3) d(Av(x),M) = mind(tTx + ( - t)x,M); 
in other words, A(x) is the unique point in aff{x, Tx} which is closest fo M. 
(3.7.4) IIAT(x)II = rein IItTx +(1 - 
in other words, Av(x) zs the unique point in aff{x, Tx} havre9 minimal no. In 
paicular, 
(3.7.5) I]AT(e)II  min{]]x]], 
ProoJ. Using Lemma 3.3, we can write 
IItTx + (1 - t)x - y[I 2 = IItTx + (1 - t)x - Av(x) + AT(X) -- yl] 2 
= II(t - t)(Tx - x) + (AT(x) - y)ll 2 
= (t- t)2[ITx- xii 2 + ]]Av() - yll , 
which proves (3.7.1). Equation (3.7.2) follows immediately from (3.7.1). Moreover, 
(3.7.3) follows by taking the infinmm over all y G M in (3.7.2). Finally, (3.7.4) 
follows from (3.7.2) by taking y = 0. 
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While AT is not linear in general, it does share some inlportant properties of the 
linear mapping PM- Nanlely, itis continuous, homogeneous, and "additive modulo 
BI". These are recorded in parts (5), (4), and (3), respectively, of the following 
lemma. 

Lemma 3.8. Let x E X and y  11I. 
(1) 
(2) 
(3) 

(4) 
(5) 

Then: 

tx+y = 
tx = tx for every a 7[: O. 
A(x+y) = A(x)+y for every n e N. In partzcular, AT(x+y) = AT(x)+y 
and AT(y) = y. 
AT(cx) = cAT(x) for «v«ry c  . 
AT is continuous. 

Pro@ (1) Ifx  M, then x+y  1I and t+ = 1 = t. Ifx  111, then x+y  111, 
and SO, 

<x + ,x + - T(x + )> <x + ,x- Tx> <x,x- Tx> 
t+ = [,x + y- T(x + y)ll 2 = il x_ Txl, 2 = i[ x_Txl. 2 = t 

using Lenlma 3.3(2). 
(2) Let c ¢ 0. Ifx E M, then cx G /I and tx 
cx  M and 
<ax, a«- T(ax)> <x,x- Tx> 
tax = = = 
II,. - T(,x)ll  Il x - Txll  

(3) When = 1, 

Ifx  M, then 

AT(X + y) = t+uT(x + y) + (1 - t+u)(x + y) 
= t(Tx + Ty) + (1 - t)(x + y) using part (1) 
=tTx + (1 - t)x +ty+(1-t)y 
= AT(X) + y. 

Now assume (3) holds for some n _> 1. Then 

A+l(x + y) = AT[Aï.(x + y)] = AT[Aï.(x) + y] 
= AT(A(x)) + y bythecasen=l 
= A+I(x) + Y; 

so the result holds for n + 1. 
(4) If c :/= 0, then by (2), 

AT(Cx) = tT(cx) + (1 - t)(cx) = t[cT(x)] + (1 - t)[cx] = cAT(x). 

Since AT(O) = 0, the result also holds when c = 0. 
(5) If x  X \/1I and x --* x, then since X \ M is open, x /I eventually, and 
SO, 
<x,x,-Tx <x,x-Tx> 
«æ,, -- IIx,, - Tx,ll 2 --' IIx - Txll 2 = t 
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and hence AT is continuous at x. Ifx (5 M and  > 0, let y (5 X with Ily-xll < e/3. 
Then IIP^tx - PMyll -< IIx- Yll < /3 and 

I1: - AT(Y)II < IIx- PMull + IIPMu -- AT(Y)II 
I1:- P^Yll + IIAT(U- PMY)II by part (3) 

by (3.7.5) 

Remark. We note that, while AT is conthmous, it is not uniformly continuous, in 
general, unlike a linear operator. For exmnple, let X = £2, let {en 
be an orthonormal basis for X, and define T on X by Tx = ,(x,e)n/(n + 1)e,» 
Setting x, = (1/n)e0 + ((n + 1)/)e, and y, = e, for all  > 1, we get that 
IIx. - ull = (v/,)  0. But nsing the readily deduced fact.s that AT(y,) = 0 
and Ar(x)= (1/2)(e-e0) for ail n, we obtain that [IAr(z)-Ar(u)l[ = (v/2) 
for al ln> 1. 

Lelnlna 3.9. (1) t >_ ½ for all x (5 X; and 
(2) Fix Af = M(= Fix T). 

Proof. (1) If x (5 M, then t = 1. If x  M, then the quadratic function, 
q(t) := IIt(Tx - x) + xii 2 = at 2 + bt + c, 
where a := [[Tx - xii 2 > 0, b := 2(x, Tx - x), and c := Ilxll 2 is strictly convex 
and attains its mininum at the unique point t when q'(t) = 0: that is, when 
t : fmi n :: __b Hence, 
2(x, Tx - x) (x, x - Tx) 
¢' = 211:: - :112 = IIx- T:II 2 =: t» 
But c = q(0) = I1:112 and IIT:II 2 = q(1) = a + b + c = a + b + II:ll 2 implies that 
-b = a + [Ixll 2 -IITxl[ 2 and hence 
-b a+ IIxll 2- IITxll 2  Ilxll 2- IITxll 2 1 
t x : fmi n : -- ._ : -- -- > --. 
2a 2a 2 2a - 2 
(2) x (5 FixAT if and only if x = txTx + (1 - tz)x if and only if t(Tx - x) = 0 
if and only if Tx - x = 0 (using part (1)) if and only if x (5 FixT = M. [] 
1 for t is sharp. To see this, take T -I and note 
Remarks. The lower bound  
that tz ---   for every x (5 X\{0}. Also,. if we relax the condition that T be 
1 for each 
nonexpansive and consider T = AI for A : 1, we deduce that tz, - _ 
x : 0. By varying A, we see that tz,M can take on every nonzero value. 
Definition 3.10. Define f = fT " X  IR + := {a (5 IR I c > 0} by 
{ ,IAT(x)--Pxll if Tx  M, 
f(x) := IITz-P»zll 
0 if Tx (5 3,1. 
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Lemma 3.11. For each x ¢ X, we have 0 <_ f(x) <_ 1 and 
(3.11.1) 
Proof. This is immediate from (3.7.2) with y = PMx. [] 
Lemma 3.12. T commutes with PM and P^p_. 
Pro@ For each x ¢ X, 
P,Tx = PaT(PMx + PMX) = PM[T(PMx) + T(PMx)] 
= Px since T(M ) C I  by Lemma 3 3(4) 
= PMx = TPa« 
Thus, T commutes with PM and, since PM = I -- PM, it follows that T also 
comlnutes with PMi. 
Definition 3.13. Let T be a nonexpansive linear operator on X, I = Fix T, and 
for any  ¢ N, let ch(T) denote the norm of the linear operator (TPM)n: 
(3.13.1) c(T) := II(TPM)II. 
ç will often write c(T) inste of ci (T). Note that if T = PMPM_ ' PMi, 
then BI := î BIi = FixT and 
(3.13.2) c(T) = IIPM P_,... PM, PMII =: c(II,h,. "'Ik) 
is just the cosine of the angle of the k-tuple (Ml, M,..., Mk) defined by Bauschke, 
Borwein, and Lewis [21. It w established in [21 that c(T) < 1 if and only if 
Mî + M +--. + M is closed. When k = 2, 
(3.13.3) c(PMPM1) = IIPMPM1PMII = c(hl,hh) = c(hh, hll) = c(PM1PM) 
is just the ordinary cosine of the angle between the subspaces hI and hIu (see [12] 
Lemma 3.14. Let T be noneansive on X and BI = Fix T. Then 
O) c(T) = lIT  - PMI[ for every n e N. In paicular, 
(3.4.1) I]Tx- PMxll 
and c(T) is the smallest constant independent of x for which (3.14.1) is valid. 
(2) IITI  c(T) IlYll for eve   I; 
() c,(r) 
(4) c(T*T)  c(T) 2 and c(T*T) = c(T)  ifFix(T*T) = FixT. In particular, if 
T = PM PMi_, "'" Ph, then 
(a.14.2) c(r*r) = c(r); 
(5) [[AT(x)- PMX[[ 
Proof. (1) By Lemma 3.12, T commutes with PM and TPM = Pa = PMT. Thus, 
c(T) = II(TPM)II = Il[T(/- PM)III = Il(T- PMPII = lIT - PMII. 
Now fix any x  X and set y = x - PMX. Then y  I  and 
IITx- PMxll 
 c(T)Ilyll = c(T)IIx - PMxlI, 
which proves (3.14.1). 
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(2) This was essentially proved during the course of proving (1). 
(3) c(T) = I[(TP^I)I[ <_ [ITP^p ll' = ci(T) '. 
(4) Let N = Fix (T'T). Since/Il = FixT* by Lemma 2.1, it follows that M C N 
and so N ± C/lI ±. Hence, since T comnutes with P^p by Lemma 3.12 and, by a 
similar proof, T* commutes with P^p, we obtain 
c(T*T) = IIT*TPN_II <_ liT*TF^pli = II(TPM. 
= IITP^.II 2 = c(T) 2. 
Moreover, if Fix(T*T) -= FixT, then N ---- M and N ± -- M ±. So the above 
inequality must be an equality. Equation (3.14.2) holds when T is a product of 
projections by Lemma 2.4. 
(5) Fix x E X. Then x - P^x  llI ±. So Lemma 3.11 and part (1) imply 
IIAT(Z) -- P^lzll = f(x)IITx- PMXii <_ f(x)c(T)llx- P^,xll. 

Renark. The following example shows that the strict inequality c(T*T) < c(T) 2 
is possible in part (4). For let X denote the Euclidean plane and define the linear 
operator T on X by Tx = x(2)e + x(1)e2 for each x = (x(1),x(2)) G X, where 
el = (1,0) and e2 = (0,1). It is ey to verify that [[T[[ = 1, I := FixT = 
span (el +e2), c(T)= 1, T= T*, T'T= I, and c(T*T)=c(I)=0. 
Lemma 3.15. Let T be noneansive and M = Fix T. Then 
(1)  T is nomal, then 
(2) ffFix(T*T)= FixT, then 
(3.15.1) Il(T'T)  - PMll = c(T) = for ever v n  N. 
In paicular, for every n  N, 
(3.15.2) I(PM, PM "'" Pf Pf_, "'" Pf,)" - PI = c(M,M2 ..... Me) . 
Proof. (1) Since T is normal and T commutes with PMz by Lemma 3.12, we deduce 
that TPAx is normal. Hence, using Lemma 3.14(1), we obtain 
[[T  -PM[[ = c.(T) = [](TP)"]] = []TPMz ]]" = c(T) . 
(2) Since T*T is selfadjoint, hence normal, apply part (1) to T*T instead of T 
using that Fix(T*T) = 
c(T*T) = c(T)  and (3.15.1) follows. 
(3.15.2) follows from (3.15.1) by taking T = PP_, ... P, and using Lemma 
2.4 to get Fix T*T = Fix T. 

The following theorem gives an upper bound on the rate of convergence of the 
accelerated scheme. 

Theorem 3.16. Let x  X and set 
x, := A-(Tx) 
Then for every n  N, 
(3.16.1) 

(n = 1,2,...). 
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and 
(3.16.2) 
Proof. The relation (3.16.1) is a consequence of Lemma 3.14(1) and (3). 
We prove (3.16.2) by induction on n. For n -- 1, 
by (3.14.1). Since t.he product of alw set. of scalars over thc cmpty set of indices is 
1 by dcfinition, (3.16.2) holds when n = 1. Now assume that (3.16.2) holds when 
,=m> 1. Then 

IIA(x)- P^xll = 

<_ f(x)c(T) 

c(T)m÷lllx- 

which shows that (3.16.2) holds with  replaced by m + 1. 
induction. 

This completes the 

Remarks. By comparig the right sides of (3.16.1) and (3.16.2), t.his result seems 
t.o suggest t.hat the accelerated algorithm is always faster t.han its unaccelerated 
at. least t.he factor [1-Iî -1 f(xi). Indeed, we will show below t.hat. 
counterpart. 
by 
J 
when T is selfadjoint, nonnegative, and nonexpansive, then the accelerat.ed method 
is faster than the original (sec Theorem 3.20). In particular, the accelerated sym- 
metric MAP is fa.st.er t.han tue symmetric MAP. Alto, t.he accelerat.ed MAP for t.wo 
subspaces is fa.st.er t.han the MAP. Perhaps surprisingly, however, we will sec that. 
t.his is not. alwws t.he case, in general, for the accelerated MAP when there are 
more than two subspaces. 
Theorem 3.16 can be strengt.heed in tue particular case when T = PuP. To do 
t.his, it is convenient to appeal to the following simple lemma (sec, e.g., [13]). 
Lemma 3.17. Let kll and h& be closed subspaces with bi = hl  h& and let Pi 
be the ortho9onal projection onto kl for i = 1,2. Then c(PP) = c(al, ah) and 
(1) gx  I 0I x, then I[Puxl[  
Proof. That c(PuP1) = c(l,,.lu) in t.his case was observed following Definit.ion 
3.13. 
(1) Let x  I  M x. Then 
(2) The proof is similar t.o (1). 
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(3) Let x  M2 ç M'. Then Plx  M1 ç M'; so by (1) and (2), we obtain 
IIP2Plxll <_ c(P2P1)llPlxll <_ 

Theorem 3.18. 

Then 

Let T = PMzP^h, x  X, and 
xn:=A--(Tx) (n= 1,2,...). 

(3.18.1) 

IIA.-l(Tz)- P^xll <_ [If(xd] c(M,Al2)2-lllx- P^asll. 

Pro@ The proof is by illduction and proceeds just as in the proof of Theorem 
3.16. The Olfly point that should be noted is that in the induction step, we use 
lhe inequality IIT(x,) - P^t(xm)ll <_ c(T)2llXm - P^(Xm)II (rather than the same 
expression wih c(T) instead of c(T)  that was used in Theorem 3.16). The proof 
of this inequality follows inunediatelv from Lemma 3.17(3). [] 

Remarks. (1) Gearhart and Koshy [13] established (a weaker version of) the special 
case of Theorem 3.18 when c := c(M, M2) < 1 and with an additional factor p on 
 >1 
the right side of (3.18.1), where p :=  _ . 
(2) The inequality (3.18.1) improves the bound on the ordinary MAP in case 
k = 2, due to Aronszajn [1], who showed that 

(3.18.2) II(P2PIFz- P^»r]l <_ c(Bl,Al)n-ll'oe - P^tasll for all as  X. 

In fact, Kayalar and Weinert [17] showed that the Aronszajn bound is sharp, i.e., 
II(PzPI) ' - P^II -- C(1[1,1[2) 2n-1 

Next we show that the accelerated algorithms are always at least as fast as 
their unaccelerated counterparts provided that T is selfadjoint, nonnegative, and 
nonexpansive. It is first convenient to establish the following result. 

Lemma 3.19. 
(3.19.1) 
then 
(3.19.2) 

// 

IIT-I(AT(as))II <_ IITxll for every x  M" and n  N, 

IIA,î,-(Tas)ll _< IIT'asll for every x  A[_L an  r. 
In particular, if (3.19.1) holds and the original algorithm converges, then 
(3.19.3) IIA-(Tx) - PMxll <_ IITnx - P^xll for evry x  X,r  N, 
and hence the accelerated algorithm converges at least as fast as the original. 
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Proof. When n : 1, (3.19.2) is trivial. If z _> 2, then for each x E 11I ±, 
IIA,-l(Tx)l[ = I[AT(A-2(Tx))[[ <_ IIT(A-2(Tx))II using (3.7.5) 
= IIT(AT(y))II, where y := A-a(Tx)  M  by Lemma 3.3(4) 
 ]]Ty]] by (3.19.1) 
= T2(A-3(Tx))[ 
: HTe(AT(z))II, where z := A-4(Tx)  M x by Lelnna 3.3(4) 
5 l[T%l by (3.re.l) 

Çontinuing in this way, we end up with the il,cquality [[A-I(Tx)II < IITxll, which 
verifies (3.19.2) when n > 2. 
To verify the last statement, let. x E X. Then x - PMx G M - and so by (3.19.2) 
and Lemma 3.8(3), we get 
IIA-I(Tx)- P^,xll = IIA-I(T(x- P^,.r))ll < IIT'(x- P^,x)ll = IIT'x- 
and this verifies (3.19.3). [] 

The natural question raised by Lemlna 3.19 is this: for which T does (3.19.1) 
hold? Vfe will show next that if T is selfadjoint, nolmegative, and nonexpansive, 
then (3.19.1) and hence (3.19.3) hold. It should be noted that our proof seems to 
use the spectral theorem (for compact selfadjoint operators) in an essential way. 
Theorem 3.20. Let T be selfadjoint, nonne9ative, and nonexpansive. Then 
(3.20.1) IIA-a(Tx) - P^,xll <_ IITnx - PMxll for «a«h x  X and n  bi. 
In other words, the accelerated algorithm converges at least as fast as its unacceler- 
ated counterpart. 

Corollary 3.21. IfT = P1P2"" PkPk-1 "'" P1, then 
(3.21.1) IIA-I(Tx) - PMXI[ _< []Tnx- PMx[I for each x  X and n  N. 
In other words, the accelerated symmetric MAP is at least as fast as the symmetmc 
MAP. 

The corollary follows since T = Q'Q, where Q = PkPk_l ..-Pl. 

Proof of Theorem 3.20. By Lemma 3.19, it suffices to show that 
(3.20.2) [[Tm-IAT(y)[[ <_ [[T"g[[ for every y  1"il ± and riz  N. 
Toward this end, fix y G M ± and m G N. If y = 0, (3.20.2) is trivial. Thus, by 
scaling and Lemma 3.8(4), we may assume [[y[[ = 1. If m = 1, then (3.20.2) follows 
from (3.7.5). Thus, we may assume m _> 2. Let. 
N = span {y, Ty, T2y .... , Trg}. 
By Lemma 3.3(4), N C M ±. Define S := PNTPN. Then S is compact, selfadjoint, 
nonexpansive, 7(S) := Range of S C N, and so n := dimT(S) _< m + 1. We may 
assume that Ty ¢ O. For if Ty = 0, then AT(y) = 0 by (3.7.5); so (3.20.2) holds 
and we are done. But if Ty  O, then Sy  0 and hence n _> 1. As a consequence of 
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the Spectral Theorem [3, Corollary 5.4, p. 47], we readily deduce that there exists 
an orthonormal set of n eigenvectors {Vl, v2,..., v,} of S such that 
(3.20.3) Sx :=  A,(x,v,)v, for every x e X, 
where Ai is the (nonzero) eigenvalue corresponding to vi " Svi = Aivi (i = 
1,2,...,). In particular, {v,...,v} is an orthonormal bis for R(S). Since 
T is nonnegative, 
Ai = i(v, v) = (Aiv, v) = (Svi, v) = (PNTPNv, V) 
= (TPNv, PNV) = (Tv,v) since v e N 
0. 
Thus, Ai > 0 for each i. Since S is nonexpansive, 
Ve have shown that 0 < A  1 for each i. Moreover, if some 
So equality must hold throughout this string of inequalities. Using the condition of 
equality in Schwarz's inequality, we obtain Tv = pv for some p > 0 and [[Tç[[ : 
[[ç[[ = 1. Hence, p = 1 and Tri = ç. That is, çi  FixT= M. But çi  fil x 
implies that çi = 0, a contriction. This proves that Ai < 1 for each i. Hence, we 
bave shown that 
(3.20.4) 0<A< 1 lori= 1,2, .... 
Let i :: (g, ç) for each i. 
« 1.  = so : ,=, m ( = 1. .... ,). 
The formula for S, SSg   
=  Açi, follows eily from (3.20.3) and the fact 
that Sçi : Aiçi. To prove the corresponding statement about T, we proceed by 
induction on j. For j = 1, since y and Ty are in N, we obtain Ty = PNTy = 
PNTPy : Sg; so the result holds when j = 1. Now suppose the result holds 
when=lm-1. Then 

St+ly : S(Sty) : S(Tty) : PNTPN(Tty) : PNT(Tty) = PNTt+ly = Tt+ly 
since Tt+ly E N. This proves the claim. 
Since 7(S) ± : Af(S*) : Af(S), where Af(S) is the null space of S, we bave that 
X : 7(S) @ Af(S) and hence we can write y as y = y + Yo, where Yl e 7(S) = 
span{vl,v2,...,v,} and Yo e span{vl,v2,...,v,} ± = Af(S). Then 
?'t ?'t 
1 1 1 

and 

2 
llvll  = ] m + llyoll . 

' c-A'-{1 - (1 - A)tv}v i. 
Claim 2. Tm-IAT(y) : -=1   
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We compute 
Tm-lAr(y) = T'-l[tTy + (1 - t)y] = tT'y + (1 - t)Tm-ly 
=  iAr-:{tv%i + (1 -- tv)}vi =   A m-: (i --(1 -- 
1 1 
which proves the claire. 
By Claires 1 and 2, we sec that (g.20.2) holds if and only if 
 '- {  -(: - a,)t}  _<  "-' 
i=1 z=l 
which, after some Mgebra, lnay be rewritten as 
(a.0.S) q(t )  0, 

where 

(3.20.6) 

Claim 3. The function h, defined on the nonnegative rem line by 
a i Ai(1 - Ai) for all t _> 0, 
h(t):= E, 2 t 
is increasing. 
Writing h(t) = u(t)/v(t), it suffices to verify that h'(t) > O. 

suffices to show that 
(3.20.7) 
Setting 

u'(t)v(t) >_ u(t)v'(t) for all t >_ O. 

n 
we see that fii E 0, -l Pi = 1, and (3.20.7) may be rewritten as 
Since the function ¢  t In t is convex on (0, oe), it follows that 
On the other hand, the function t  ln is concave on (0, oe); so 

Equivalent ly, it 
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Combining (3.20.9) and (3.20.10), we obtain (3.20.8), and this proves Claire 3. 
To prove (3.20.5), and finish the proof of the theorem, we tnust verify that 
q(tg) < 0, where q is the quadratic defined in (3.20.6). Now q(0) = ? > 0 and 
q(1) = c - 2fi +  = 0. Also, an inspection of the coefficients shows that 0 < a < 
 < . Further, the quratic forlnula shows that the zeros of q are given by 
Since  = }(a + ), it follows that ti = 1 and t = /a > 1. Su,ce q h a 
positive leading coefficient, we see that q(t)  0 if and only if train < t < tmax, i.e., 
1 < t < 7la. Thus to prove q(t)  O, we must show that 
lt ? 

(3.20.11) 
We have, using Claire 1, that 

(3.20.12) 

But 
(3.20.13) Eiaî(1 - Ai) + Ily0ll 2 _< Eic/(1 - Ai) 
E(1 - )= + I0], = E( - )= 
eonow sce E ( - ) e E ( - )=- 
By Claire 3, h is increasing so that h(0)  h(2m - 2). That is, 
(3.20.14) Ei 
E (1- )= - E-='=-=( - j)= 
Combining (3.20.13) md (3.20.14), we obtain (3.20.12) and hence t9  /a. This 
proves (3.20.12), and completes the proof of the theorem. 
A certain analogue of Theorem 3.20, valid when T is hot selfadjoint, can be 
deduced from Theorem 3.20  follows. 

Corollary 3.22. Suppose S is a bounded linear operator on X, L is a closed 
subspace of X such that L D Tt(S), and SPL is selfadjoint, nonnegative, and non- 
expansive. Let/lI = Fix S. Then 
(3.22.1) A'-'P 
  Lx--PMXlI<_ IIS'PLX--PMxll foreachxGX andzzGN. 
In particular, 
(3.22.2) IIA-Sx - Ptxll <_ IIS'x - PMxll for each x  L and zz  N. 
Proo]. Set T = SPL. Then T satisfies the hypothesis of Theorem 3.20. Moreover, 
since T/(S) C L, it follows that FixT = FixS = M. Thus, we deduce from (3.20.1) 
that 
(3.22.3) ]]AT-I(Tx)- PMxll <_ IITx- PMxll for each x  X and n  N. 
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Since Te(S) c L, we deduce that 
T n = (SPL) n - S(PLS)n-IpL-S(s)n-IpL :S PL.n 
In particular, Tn = Sn for each x E L. Moreover, for each y E L, 
AT(y) = ty,TTy q- (1 - ty,T)y = ty,TSy q- (1 - ty,T)y 
and Asy = ty,sSy + (1 - tu,s)y. But 
so AT(y) 
= .a s y. Substituting back 
into (3.22.3), we obtain (3.22.2). In general, for any y  X, x = PLY  L, and so 
(3.22.4) [IA-ISpLU - 
But M C Te(S) C L; so PMPLU = PMU and substituting this into (3.22.4) yields 
(3.22.1). [] 
One application of Corollary 3.22 is in the case of the MAP for two subspaces. 
Theorem 3.23. Let ]il and ]il be closed subspaces in X, Q = PP1, and 
1lll V ]il. Then for each n  l, 
(3.23.1) [IAQ-1Qx- PM][ _< [IQ'x- PMx[] for every x  X. 
In other words, th.e accelerated MAP is faster than the MAP in the case of two 
subspaces. 
Proof. Take S = Q and L = ]ile in Corollary 3.22 to obtain 
(3.23.2) I[AQ-1Qpx- PMx[[ <_ I[QPx- PMx[] for every x  X. 
In particular, (3.23.1) holds for each x  ]il» It remains to show that (3.23.1) holds 
for all x  X. ,Ve first verify 
(3.23.3) Te( PPI P) = Te( PP1). 
To see this, note that it is well-known that for any bounded linear operator T on 
X, 
(3.23.4) .N'(T*T) = Af(T) and A/'(T) J- = Te(T*). 
Putting T = PP in (3.23.4), we obtain that .N'(PP1P) = .N'(P1P) and hence 
Te( PP1P) = JV'( PP P) ± = .N'( P1P) ± = Te( PP ), which proves (3.23.3). 
Now fix any x  X and set z = PPlx. Then z  Te(PP1) and so, by (3.23.3), 
we obtain that z = lim zk, where zk  Te(PP1P) for each k. Then we choose 
wk  X so that zk -= PPIPWk. Let Yk :: IP2Wk. Then Yk  _1"li2 and zk = P2PlYk. 
Since P commutes with P^ for i = 1, 2 we bave that 
PMx = PP1PMx = P^tP'2Plx = PMz = lira PMz 
k 
(3.23.5) = lira PMPPlyk = lira PPIP^.yk = lira P^y. 
k k k 

Moreover, 
(3.23.6) 

lira QYk = lira PPlyk = lim zk = z = Qx. 
k k k 
By (3.23.2) applied to yk  1lI, we obtain that 
(3.23.7) -1 
I[AQ Qy- P^yll <-IIQy- PMyII. 
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Letting k -- oc in (3.23.7), and using (3.23.5), (3.23.6), and the continuity of AQ 
(Lemma 3.8(5)), we obtain (3.23.1). [] 
The following is an example showing that the accelerated MAP may be slower 
than the MAP when there are more than two subspaces! 
Example 3.24. Let X = g2 and let ci (i - 1, 2,...) denote the canonical unit 
vectors in X: ci(j) = 5i. for all i, j. Define rive 2-dimensional subspaces as follows: 
M1 = spn {2,3}, M = spa {« + «,a + e}, Ma = p {e.e}, 
M4=span{el+«,e+es}, and M5=span{e,,e2}. 
Let & = P,, for i = 1, 2 .... ,5 and T := PsPP3P2P. It is ey to veri that 
Tx = à£()«l + à£(a)«2 for ea«h x 
1 1 
Also, [[T[ =  and M := FixT = {0}. Set xo := 4e3. Then TXo = e2, T2xo = e, 
and Txo = 0 for all n  3. 
Let z0 := Txo = e2 and define Z := AT(Z-I) = A(Zo) for n  1. Since 
thc range of T is span {e, e2} and AT(x) is an affine combination of Tx and x, it 
follows that 
(3.24.1) z = ae + ee (n = 0, 1,...) 
for some scalars a, fl.  will prove that z ¢ 0 for every 
Having donc this, we would then obtain for every n  3 that 
whi«h shows that the accelerated MAP is slower than the MAP beginning with the 
third iterate. (It should be noted, however, that the second iterate for the accel- 
erated method h a strictly snmller norm than the corresponding unaccelerated 
term: [[AT(TXo)[[ = 1/ < 1/4 : [IT2xol[.) 
It remains to show that z ¢ 0 for each n, and this will be donc through a series 
of claires. 
Claire 1. Tzn = çnel (n = 0, 1,...). 
This follows from 
Tzn = T(ael + e2) aTe + nTe2 
Next we prove 
Clazm 2. zn+ = 0 if and only if n = O. 
For suppose zn+ = 0. Then 
0 = AT(Zn) = tnTzn + (1 - tn)Zn, where tn = tz, 
 ( t.)(. ) 
= te + - + 
: [kV.t. + ( - t.)..]  + ( - t.)Z.» 

It follows that 

¼,t, + (1 - t,)c, = 0 and (1 - t,), = 0. 
No marrer what the value of tn is, the two equations above imply fl, = 0. 
Conversely, if fl = 0, then z, = c,el implies that Tz, = 0 and hence AT(z,) = 
0 (since IIAr(zn)ll < IITznll by (3.7.5)). Thus, zn+l = AT(zn) --- O. 
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Claim 3. For n -- 0, 1,2,..., 
(3.24.2) /3+1/3 = +1 (41-/ - ). 

In particular, if/3 ¢ 0, then 
(3.24.3) /3+1 -- On+l 4 " 

To verify this, note that by Lemma 3.3(2), we obtain that 
<z,+,z, - Tz,) = <AT(z,),z, -- Tz,) = O. 
Using the representation of zn in (3.24. I), we expand the above equation and deduce 
that a+l(a - -àfl) + fl+lfl -- 0, which is just (3.24.2). 
If the result that z ¢ 0 for each n were false, we let no denote the smallest 
integer such that z,o+l = 0. Now/o -- 1 and one can readily compute that 
zi = AT(Zo) = AT(e2) = te2Te2 + (1 - te)e 1 
where 

(e2, e2 - Te2) 1 16 

Hence, z1 7el-- 7e2 5ndso oQ - 7 5nd j 1 1 
= = . Thus o ¢ 0 and 1  0. 
By Clm2, no = 0; sono  2, andn ¢ 0 foreveryn  no- 1. rther, by 
Claire 3, we deduce 
(3.24.4) nl : nl( - #n) for n : 0, 1 .... ,no - 1, 
where pn := an/fin. 
From (3.24.4), we deduce that an+l ¢ 0 whenever n+l ¢ 0 and 0 g n g no - 1. 
Since k+ ¢ 0 for 0  k  no - 2, it follows that ak+ ¢ 0 for 0  k  no - 2. In 
other words, 
(3.24.5) 0 and 0 for 1 nno-1. 
Using (3.24.4), we obtain that 
(3.24.6) 0 ¢ n+l 1 
+ 4 p for 0n<n0-2. 
Next consider the fo]]owing subset of the rationa] numbers: 
P 
Q*:={ [p, qeZ, peven, qodd}. 
I 
In particular, 0  Q* but   
Claire 4. The function (x) = ( - x)-I maps Q* into Q*. 
1 Q.. Q.. 
First, note that f is we]]-defined since   Next, let x  Then x = q 
for some even p and odd q. Hence, 
1 4q 
() = 1 e q_ ap 
4 q 
Since 4q is even and q - 4p is odd, it follows that f(x)  Q*. 
1 for 0 <n <n0 I. 
Claire 5. pn  Q* for 0 g n  n0 - I. In particu]ar, pn ¢  _ _ - 
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 -- 0  Q*. By (3.24.6), it follows that 
To verify this, first note that Po = fo - 
n+l 1 
-  (n = 0, 1,...,no- 2). 
(3.24.7) 
Using (3.24.7), Claire 4, and induction, it follows that p+  Q* for n : 
0, 1 .... , n0 - 2. This proves Claim 5. 
 from Claire 5. Since no = 0, (3.24.4) implies that ano = 0. 
Finally, Po- 
But then z o : oe + noe2 = O, which contradicts t}le choice of n0- This proves 
that the accelerated MAP is slower than the MAP for this example. However, both 
the MAP and the accelerated MAP do converge[ This raises an interesting question 
that we pose now. 
Open Problem. Let T be a nolmxpansive mappilg on X which is ymptotically 
regular, and let M = Fix T. Then, by Corollarv 2.3, the algorithm converges: 
(3.24.8) lira IITx - PMxll = 0 for ea«h x G X. 
Is it true that the accelerated algorithm for T also converges? That is, does the 
following hold: 
(3.24.9) lira IIA(T.r) - PMxll = 0 for each x e X? 
We bave seen that the answer is amat{ve in several special ces. For example, 
when any Olm of the follow}ng condit}olm are satisfied then (3.24.9) holds. 
(1) T 
(P P_, "" Pt, )* (P P_, "" Pt, ) (Crollary 3.21 ). 
(2) T = PPM is the product of two orthogonal projections (Theorem 3.23). 
(3) c(T) < 1 (Theorem 3.16); iii particular, if T = PMP_ ""Ph and 
I +  +... + I: i ¢od, t. c(T) < 1 ( [e]). 
In particular, does (3.24.9) hold if T is the product of k k 3 orthogonal projec- 
tions? In this ce, we can show that 
(3.24.10) A(Tx)  PMx weakly for each x E X. 
But we are not sure whether the convergence must be in notre. 
To prepare for the last main result, we begin with a useful lemma. 
Lemma 3.25. Define the function 
E(a,).- 2-a- foralla,E witha+#2. 
(1) Then E is a continuously differentiable function on its domain such that 
OE(, ) 2(- 1) 
(a) O (2 -  - )' d 
(b) 0(,) _ 2(1 - 
0Z (2 - a - )" 
In particular, ff c  1, then E(a, c) (respectively, E(c,)) is a decreasing 
(ptiwty, i««ig) Icto ol  (p«ctiwty, Z) i ch ol th two 
components of ifs domain. 
(2) (a) I«(',[ < 1 ffnd oty iI(1-)(1- > 0. 
(b) «(,, = 1 ffnd oty I(1-,)(1-) = 0. 
(«) «(,) >  ffd otv i/(1- ,)(1- Z <0. 
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Pro@ The verification of (1) is easy. 
(2) Write 

p-a (l-a)- (l-p) r I -r2 
E(o,/3) -- 2- o-/3 (1 - o) n t- (1 --/3) rl n t-r2 

where ri -- 1 - a, and r2 = 1 -/3. Clearlv, [E(a,/3)[ < 1 if and only if [rr__=[ < 1 
- - ri -l-r2 
if and only if [ri -- r21 < ]ri + r2] if and only if rit2 > 0. This provcs (a). Also, 
]E(,fl)] = 1 if and only if tir2 = 0, which proves (b). Finally, IE(,fl)] > 1 if and 
onlv if ri 
 + > 1 if and only if 

Lemma 3.26. Let T be selfadjoint, 
Cl := inf{(Tx, x) [ x  Iii ±, [[x[[ = 1}. 

(3.26.1) 
and 
(3.26.2) 

c :=sup{(Tx, x) Ix I ±, [[xii = 1}, 
where both c and c are defined fo be 0 if M j- = {0}, i.e., if M = X. Then 
(3.26.3) lnax{c2,--CI } = c(T) := iITP«" II. 
Moreover, if T is also nonnegative, then 
(3.26.4) c2=c(T). 
Pro@ First note that 
-Cl =-inf{(Tx, x)]x  5I , ]]x]] = 1} =sup{-(Tx, x)]x G M z, ]]x[] = 1}. 
Hence, 
max{c2,-«l} = sup{[(Tx, x)[ [ x e M , [[x[[ = 1} 
=sup{'(TPx,PIx)I I x  X IIxll = 1} 
(using Lemma 3.12 and the idempotency of PM ) 
(since TP is selfMjoint and using [3, Proposition 2.13, p. 34]) 
= (), 
whieh proves (3.26.3). Finally, if T is also nonnegative, then 0  Cl  c and so 
max{c2,-c:} = c2. Thus (3.26.4) follows ffoln (3.26.3). 
Lemma 3.27. Let T be selfadjoint and noneansive, and let c and c2 be defined 
as  (3.26.1) and (3.26.2). 
- 2 - Cl 2c2 I[yli for eve y  M . 

(3.27.1) 
In particular, 
(3.27.2) 

( )n 
IIA()II <_ 2- C I -- C 2 

for every y  Al ±, n  N. 
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The inequality (3.27.2) follows from (3.27.1) by induction, using the fact that 
AT(y) e M ± whenever y e M ± (Lemma 3.3(4)). Our proof of (3.27.1), just like 
that of Theorem 3.20, uses the spectral theorem. Before proving this lemma, let us 
state a few consequences of it. 

Theorem 3.28. Let T be selfadjoint and nonexpansive, and let cl and c2 be defined 
as in (3.26.1) and (3.26.2). Then 

/ 
(3.28.1) 
- P^«xll _< [, 
for every x 6 X and   N. 

2 - cl - c2 

Proof. Let x  X and set y = Tx- PMx. 
Substitute this y into (3.27.2) (and replace  by n - 1) to obtain 
)n--1 
IIA-( Tx - P^¢x)ll _< ( 2 -9 ciZ zc c2 IITx - PMxlI. 
But A-(Tx- PMx) = A-(Tx) - PMx by Lemtna 3.8(3) and 
IITx- Pa, xll = I[T(x- PMx)[I  c(T)llx- P,xll by (3.14.1). 
This proves (3.28.1). 

Then y e M ± by Lemma 3.3(6). 

Theorem 3.29. Let T be selfadjoint, nonnegative, and nonexpansive. Then 
c(T) " 
(3.29.1) ]IA,-(Tx)-PMxI[ < [2_ c(T)]_  I[x--PMxll for every x  X, n e N. 
Proof. Since T is nonnegative, c k 0 and c(T) = c2 by Lemma 3.26. Since T is 
nonexpansive, c2 < 1. Thus 
0 < c < c2 =c(T) < 1. 
Then, using Theorem 3.28, we obtain that for every x 6 X, 

(3.29.2) 

I]Aî-(Tx) - P^,ll < c2- c, c(T)llx- PMxl] 
-- 2 -- C I -- C 2 
( c(T) - c ) n- 
= 2 :- J (T) c(T)llx- PMxII. 

Now c(T) - cl 
2- c -c(T) = E(c,c(T)), C I and 0 are in the sanie component of the 
domain of E(., c(T)), and E(-, c(T)) is a decreasing function by Lemma 3.25. This 
implies that 
c(T) -c c(T) 
2-cl - c(T) : E(«,c(T)) < E(O,c(T)) - 2-c(T)" 
This together with (3.29.2) yields (3.29.1). [] 

Remarks. Çomparing (3.29.1) with (3.16.2), we see that for each selfadjoint, non- 
negative, and nonexpansive operator T, it follows that 
(3.16.2) [,A-(Tx)- PMx,, < [f(xi)] c(T)n,,x- PMx,, 
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and 

c(T) n 
(3.29.1) [[A-I(Tx)- P.xll  [2_c(T)]n_ 11[x- PMII. 
Thus it is natural to ask whether one of these bounds is always better than the 
other. In other words, do either one of the following two inequalities always hold: 

n--1 
(a) H f(xi) _< [2- c(T)] n-1 for ail n _> 2, 
1 

or 

n-1 
(b) 1 
< H f(xi) for all n >_ 2? 
[2- c(T)] n- - 1 

We now show that neither of these two inequalities always holds. To see that 
inequality (b) does not always hold, consider the example when X = g2(2) is 
the Euclidean plane, /iii (resp., M2) is the horizontal (resp., vertical) axis, and 
T = P^t, PMPM,. Then T = 0, _M = FixT = {0}, c(T) = I[TP^t[I = O, f(x) = 0 
for all x E g2(2), and 1 1 Hence 
2-c(T) -- 2" 

H f(oe) < [2 - c(T)l  
1 

for every n k 1. 

To see that (a) does not always hold, let X = g2(2) denote the Euclidean plane 
and define T on X by T(Cel +/3e2) -- ce + A60/3e2. Then T is a nonnegative 
selfadjoint linear operator on X, M = FixT = {0}, and c(T) = IITII = 1-60" Letting 
10 10 9 1 
x0 := iïel + ïe2, we can easily deduce that xl := Txo = y6el + y6e2, Txl 
19 (x,,x-Tx,) mo and AT(Xl) = tx, Txl + (1 - tx,)Xl = 
ïooel + ]-6-6-6e2, txl -- [ixl_Txll2 -- 41 ' 
36 IIAT(Oe)II = 1000[ 656 )½ = 0.9913034925--- and 
el -- he2. Hence, f(xl) -=-- IITxll 41 397121 
1 100 _ 0.9900990099--- implies that 
2-c(T) -- 101 -- 

1 
< f(xl); 
2 - c(T) 

so (a) fails for n = 2. 

Proof of Lemma 3.27. We should first note that ci + c2 < 2, and hence the expres- 
sions on the right side of both (3.27.1) and (3.27.2) are well-defined. For otherwise, 
ci = c2 = 1 and (x, Tx) = 1 for allx E M ± with ]lx[[ = 1. By the condition of 
equality in the Schwarz inequality, this implies that x = Tx for all x G M ±. That 
is, M ± C M and so M ± = {0}. But this implies that ci = c2 = 0, a contradiction. 
It follows also that E(cl, c2) _> 0. 
In the notation of Lemma 3.25, we must show that 

(3.27.2) 

IIAT(Y)ll  E(«,c=)Ily]l for every V <E AI ±. 

If M ± = {0}, then (3.27.2) is obvious; both sides are in fact 0. Thus we can assume 
M ± ¢ {0}. Fix any y G M ± \ {0}. By scaling and Lemma 3.8(4), we may assume 
I[Yll = 1. Let 
N := span {y, Ty}. 
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Then N C 1I - by Lemna 3.3(4) and 1 _< dira N _< 2. If dira N = 1, then Ty = oy 
for some scalar c -fi 1 and thus 
0 e sp,, {} = sp {, T} = ff{, T} 
implies AT(y) = 0 since AT(y) is the point in aff{y Ty} having minimal norm by 
Theorem 3.7. Hence, (3.27.2) holds and we may therefore sume that dira N = 2. 
In particular, Ty  span {y}. 
Define the operator S := PNTPN. Then S is a compact selfadjoint (nonexpan- 
sive) operator with (S) C N, and thus n := dim(S)  2. But both y and Ty 
are in N; so 
Sy = PNTPNy = PNTy = Ty 
implies that Ty 6 (S) and hence 1  n  2. By the spectral theorem [3, Corollary 
5.4, p. 47], there exist an orthornormal bis {e}ï of (S)x(= ri(S)) and scalars 
{A}ï such that 
(3.27.3) Sx =  A(x, e>e for every x 6 X. 
1 
In particular, 
(3.27.4) Se 
so each ei is an eigenvector of S with eigenvalue Aj. Also, 
(3.27.5) = 
since each ej 6 (S) C N. Since N C AI , this proves that 
(3.27.6) c  Aj  c2 (j = 1,...,n). 
We consider two ces. 

Case 1. ri=l. 

Then since 
N = ri(S)  In(S) l n 
dimN --- 2, and dim7(S) -- 1, it follows that dim[7(S)- ç)NI -- 1. Hence we can 
choose e2 E ri(S) ± çN with [le2[[-- 1 and define A2 =0. Then (el,e2} is a basis 
for N, and Se2 = 0 = A2e2. If follows that (3.27.3)-(3.27.6) hold with n-- 2. 
Case 2. n-= 2. 
Then (S)= N and (3.27.3)-(3.27.6) holds with n-- 2. 
Thus each case can be reduced to the case when  -- 2. 
If E(cl, c2) >_ 1, then (3.27.2) is obvious since then 
for each x, where (3.7.5) was used for the first inequality. Thus, we may assmne 
that 0 _< E(c, c2) < 1. By Lemma 3.25, this is equivalent to (1 - c)(1 - c2) > 0. 
Thatis, either 1-c »0and 1-c2 »0, or 1-c < 0and 1-c2 < 0. But the 
latter inequality implies c2 » 1 which contradicts the nonexpansiveness of T. Thus, 
we must bave 1 - c > 0 and 1 - c2 » 0. That is, 
(3.27.7) -1 _< cl < - _< ce < 1 (j = 1, 2), 
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where the lower bound ci _> -1 is also a consequence of the nonexpansiveness of T. 
Moreover, since {el, e2} is an orthonormal basis for N and since y and Ty are in 
2 
N, we have//= E1 aiei and Ty = Sy = _,ï ),iaiei, where ai := (y, ei) (z = 1,2). 
Then by (3.5.3) and using the fact that aî + aoe = Ilyll oe = 1, we deduce that 
]]AT(y)]] 
Ily- 
= 
Il 
= _ [î (- )1 
î(1- ) 
Putting the expression on the right over a conlmon denominator, expanding, and 
simpli[ving, we obtain 
î( - 
(3.7.8) IIAT(Y)II = î( - 1)  + g(1 - )" 
If  = , then (3.27.8) implies that ,4r(y) = 0 and (3.27.2) is obvious. Thus we 
may assume that A ¢ A» In fact., by reindexing if necessary, we may sume that 
<» Defineh:[0,1]Nby 
(1 - t)( - ) 
(3.27.9) h(t) := 
(1 - ,) + (1 - )(1 - ) 
Sin î +  = 1, w  tht (3.7.8) inpli tht 
(3.27.0) IIAT(Y)II   {h(0  0 <  < }. 
Buth(0) =h(1) =0andh(t) > 0forall0< t < 1. Hencethemaximumofh 
over [0, 1] oeeurs for some t  (0, 1) that satisfies h(t)  0. Differentiating h and 
expanding, we deduce that 
[ta  + (1 - «)blh'(t) = (a - b)  [t(b- a) - b] [t(a + b) - b], 
where 0 < b := 1 - e < 1 -  =: a. Hence h'(t) = 0 if and only if t = bi(b- a) < 0 
o « = V/(a + 
t = b/(a + b). Thus 
Combining this with (3.27.10), we obtain that IAT(y)II   E(, ) or, equiva- 
lently, 
(3.27.11) IIAT()II 
By Lemma 3.25, E(., a) is a decreasing function so that by (3.27.7), we get 
(3.27.12) E(a,a)  E(c,a). 
On the other hand, by Lemma 3.25, E(c, .) is an increasing function. By (3.27.7), 
it follows that 
(3.7.13) E(c, a)  E(c, c). 
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Combining (3.27.11)-(3.27.13), we obtain 
(3.27.14) ]]AT(y)]] < E(el,e2), 
and this is just (3.27.2). [] 
Remarks. It is perhaps worth noting that the inequality (3.27.2), and hence the 
main inequality in each of Theorems 3.28 and 3.29, is sharp, at least for a large cls 
of operators T. More precisely, one can prove the following result. I( T  X  X is 
selfadjoint, nonexpansive, bas finite tank. and is hot the identity, then there exists 
c= - c, II*ll for , = 0, a, 2,.... 
Out proof of this result w divided into two ces: when (T) # X and when 
(T) = X. Since the proof w somewhat lenhy, we have omitted it. 
Finally, we should mention that there are examples of expansive, oelfadjoint, and 
positive mappings T for which the algorithtn (3.1.3) diverges for every nonzero x, 
but the accelerated counterpart (3.1.4) converges That is, it is hot alwoEvs neceary 
to have the original algorithm converging to be able to accelerate it. 
For example, let X be the Euclidean plane g2(2) and define T " X  X by 
T = 3z(1)e + 4x(2)e2. Then T is selfadjoint and positive, M := Fix T = {0}, 
and ]]T]] = 4 (so T is expansive). However, ]]Tx]]  3]]z]] and ]]A(Tx)]  
3-+]]x]] for every z. This shows that ]]T"x- PM]]  oe for each z # 0, while 
IIA-a(T) - P,]I  0 for each  
Added in proof. Recently, there h been related work that h appeared since 
this paper w first submitted to the ansact.ions in July of 1999. 
First, the authors of this paper showed that the iterates 0 = ,  = A(T_) 
for n  1 generated by the accelerated map for a linear nonexpansive map T 
converge weakly to PFi() (Fejdr onotonicity and weak convergence of an ac- 
eelerated method of peetions, Canadian Math. Soc., Conference Proceedings, 
27(2002), 1-6). This generalizes t.he relation (3.24.10) above. 
F. Deutsch ( Aceeleratin9 the convergence of the method of alternatin9 projections 
via a line seareh: a brief survey, in Inherently Parallel Algorithms in Feibility 
and Optimization and their Applications (edited v D. Butnariu, Y. Censor, and 
S. Reich), 2001, Elsevier Science, 203-217) gave a survey of line search methods for 
accelerating the convergence of the method of alternating projections. 
J. Xu and L. Zikatanov ( The method of alternatin9 projections and the method of 
subspaee correetions in Hilbert spaee, J. Amer. Math. Soc., 15(2002), 573-597) gave 
an identity for estimating the norm of a product of nonexpansive linear operators 
on a Hilbert space. 
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ANDERSON'S DOUBLE COMPLEX AND GAMMA MONOMIALS 
FOR RATIONAL FUNCTION FIELDS 

SUNGHAN BAE, ERNST-ULRICH GEKELER, PYUNG-LYUN KANG, 
AND LINSHENG YIN 

ABSTRACT. We investigate algebraic F-monomials of Thakur's positive char- 
acteristic F-function, by using Anderson and Das' double complex method of 
computing the sign cohomology of the universal ordinary distribution. V'e 
prove that the F-monomial associated to an element of the second sign coho- 
mology of the universal ordinary distribution of Fq(T) generates a Kummer 
extension of some Carlitz cyclotomic function field, which is also a Galois ex- 
tension of the base field IFq(T). These results are characteristic-p analogues of 
those of Deligne on classical F-monomials, proofs of which were given by Das 
using the double complex method. In this paper, we also obtain some results 
on e-monomials of Carlitz's exponential function. 

0. INTRODUCTION 
In [Anl] Anderson invented a remarkable method of computing in an identical 
way the sign cohomology of the universal ordinary (|istributions, both for the ra- 
tional nmnber field and a global function field. He introduced a certain double 
complex which is a resolution of the universal ordinary distribution. This double 
complex enabled him to construct canonical basis classes of the sign cohomology. 
Das [Da] used this double complex in the rational number field case for the study 
of classical F-monomials and got a series of results, which greatly illuminated the 
power of Anderson's method. 
In this paper, using Anderson's double complex and following Das" method, 
we study F-monomials for rational flmction fields. Thakur [Th] defined the F- 
function in characteristic p and showed that it has many interesting properties 
analogous to the classical F-function. Especially, it satisfies a reflection formula 
and a multiplication formula. Sinha [Si] used Anderson's soliton theory to develop 
an analogue of Deligne's reciprocity for flnction fields. In the course of this he found 
that certain F-monomials generate Kummer extensions of cycltomic flmction fields, 
a result which will be reproved below with the aid of the double complex. Using 
F-monomials we also find extensions of cyclotomic function fields, and these happen 
to be Galois even over the basic rational flmction field. 
We would like to emphasize the following technical points: Besides the double 
complex, there are several main ingredients in computing the F-monomials in Das' 
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paper, and these are used frequently. In the case of a rational function field there 
are more roots of mfity, which causes the definitious of the vertical shifl, operator 
and "canonical lifting operator" tobe more complicated. In addition, the reflection 
forlmlla and the umltiplication formula of the F-function play important roles in 
out study. These forlnulae in the function field case have some extra factors, and 
thus one bas to be more careflll in applying them. 

1. THE DOUBLE COMPLEX FOR 
Let K = IFq(T) and .4 = ]Fq[T], the rational function field and polynomial ring, 
respectivcly, over the finite field Fq.  fix a generator 2 of J = Fq. Let  be the 
free abelian groul) generated by symbols lai with a  N/A. Let U be the quotient. 
of M by the s,,bgroup generated by ail eleu,cnts [a] - E.=.[bl, whr, i a onic 
polynolnial in A. and U-(resp. +) the quotieut of M by the subgroup generated 
1)y ail elements [al- E.=.[bl, along with ail the E0[0a] (resp. lai- [al). We 
call the group U the universal ordinary distribution on K/A. Further, J acts on 
U in the natural way. Let H*(J, U) denote the sign cohomology group for U. It is 
kuown that tor(U +) oe H'(J, U) and tor(U-)  H2(J,U)([BGY], Proposition 2.4). 
If a =  ,n [a]  A rcprescuts an element in H* (,l, U), we often write a  H*( J, U). 
It is clear from the context whether elelnents of A, U, H  (J, U), or H2(J, U) are 
intended.  use gothic letters to denote elements of A. Define 
 {1, ifaismonic 
( )= 0, otherwise, 
aulniug that deg a < deg [ and that [ is lonic. For a =  m [a]  A we define the 
total sure TS(a) and mteal sure IS(a) of a by E m and by IS(a) 
respectively. Let  be the least comlnon multiple of the denominators of the a and 
let t G (A/I)*. We define a  by 

Let 7' be the set of all monic irreducible polynonfials in A. \Ve fix a linear order 
"<" on 7'. Let 
$ = {la, 9, n]:a C K/A, 9 a squarefree monic polynonfial, n an integer}. 
re denote by [9 the nmnber of mouic irreducible polynonfials dividing 9. We define 
a double complex  as follows: m, = the free abelian group generated by the 
sylnbols la, 9, ni G  with m = [9]. The chain maps 0 and  of bidegree (-1, 0) and 
(0,-1), respectively, are defined by 
Il 
0[,,l = (-)-'([,1,]-  [,/,1), 
i=1 p,ba 
where 9 = p --  Pro with p < pi for i < j, and 
6[..,1 = (-)v"-[-. - - 1, fr, 
(-1)([a, 9, n - 1] -[?a, 9, n- 1]), for n even. 
Then it is easy to see that 
0 2=0, 2=0 and 0+0=0. 
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Let (T(NK), c3 + 5) be the total complex of giN. We use the same notation 
for the total complex when the meaning is evident 
Let gK' be the subcomplex of glK generated by the elements ff(a, n)[a, g, n], 
where 
-l, ifa=0andniseven, 
ff(a, n) = 1, otherwise. 
Then following the method elnployed by Ouyang in [Ou], we have: 
Proposition 1. Let U be the universal ordinary distribution on Iç/A. There exist 
canonical isomorphisms 
H2(J, U) = Ho(Ho(K,O),5) = H0(K, O + 5) = Ho(K/K',O + 5) 
and 

HI(j, 113) = H_I (H0(a]I, 0), 5) = H_I(K,O + 5) = H_I(K/K',O + 5). 

Since 
(-1)n(q - 1)[0, g,. - 1], if n is odd, 
5([0, g, n]) =  0, if n is even, 
and ô([0, g, n]) lies in K', we have: 
Proposition 2. Given a square-free monic polynomial g with [g[ = i, we define 
{ ' 
kg = [0, g, -il  glKi,_/glK,_, for i even, 
< [0, g,-i - 1]  SKi,_i_I/SK;,_i_I, for i odd. 
Then the collection {kg; Ig] even} (resp. {kg; Igl odd}) ]orms a Z/(q- 1)-basis for 
H0(SK/SK', 0+5) (resp. H_I(SK/SK', 0+5)). The kg are referred to as canonical 
basis classes. 
Define the vertical shift operator S " Km,, " ;Km,,+l by the rule 
/la, g, n + 1], if n is even, 
S([a,g,n]) = (-1) Igl [_ï___îi[7ia, g.n + 1], if n is odd. 
and define the diagonal shift operator A : Km,,  Km-l,,+2 associated with 
a prime p by the rule: 
A([a, g, n]) = O, if p ( g. 
and if g = PlP2""  Pro with Pi < P2 < "'" < 
A([a, g, n]) = (-l)"[a,g/p.,n + 2]. 
The reader can check directly the following lemma, or refer to [Da, Thms. 4-5]. 
Lemma3. i) $5+5S =q-1 andOS+SO=O. Thus (O+5)S+S(O+5) =q-1. 
ii) 0Ap = Ac3 and 5A = 
Given a canonical basis class [0, $,-n] with ]g] = n even, one tan construct a 
representing cycle 
c= (c,_,, c,_n = [o,o,-], 
i=0 
such that C,_ =  nj[aj,gj,-i] with sgn(aj) = 1 for i odd, sgn(aj) ¢ 7 q-2 for i 
even, and no terre of the form [0, -mi except [0, g,-ni occurs, as follows. 
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Suppose that one has constructed Ci.-i. If OCi,-i = j mj[a i, 9j, -il, then 
(_l)i_ 1 lE j mj(a}[a3, 9, 1- il, if i is even, 
Ci-l,l-i 
j "itj kO(a)-I [@-(«)a/, g, 1 -- il, if i is odd, 
where sgn(aj) = («) with 0 N n(aj) < q - 1. 
We call C a semi-canonical lifling. Such a construction also works for canonical 
bis classes of H 1 and for the boundary elements of 
For ml element Cof NN and a square-free monic polynonlial 9. we let C {/ be 
the 9-conlponent, i.e., the part that includes those of the form [.. 9, *]- Following 
the saine lines  Proposition 7 of [Da], we have: 
Proposition 4. Let C = ,+3=t Cid be a cycle in . For a fixed monic square- 
froc polynomial , writ« C {°t - E ni[ai, O, e - k] Thon w« bave 
k,t--k -- " 
H(J.U) gt-k is odd, 
The assertions in thc next proposition arc the analogues of Theorem 8 and Prop 
sitions 3 and 4 in [Da]. The i(le of the proof are taken from there. 
Proposition 5. Let a=  m[a]  H'2(J,U). We bave 

i) For each 
il) Let m be 
iii) Let C be 
and (0 + 

Then  ni deg pi  0 

e (A/f)*,  mi(ai) = 
the coefficient of [0] in a. Then q - 1 divides  ,ni - m. 
cycle zn  such that C = +j:0 Ci,j, Co.o =  mi[ai, 1,0], 
5)C = O. Let further 
Cl,_ 1 =  nj[bj,pj,-1]. 
mod (q - 1). 

2. ALGEBRAIC GAMMA MONOMIALS 

Thakur ([Th]) defined some F-function in characteristic p. We change Thakur's 
definition slightly by the formula 

( __)--1 
F(z) : 1-[ 1 + , 

aA+ 
where A+ is the set of ail monic polynomials in A. This F(z) is just the l-I(z) 
of Thakur. Let  denote the fundamental period of the Carlitz module which is 
unique up to a factor of IFq. Let e = ec be the Carlitz exponential. The F-function 
has the following nice properties. 
Theorem 6 ([Th], Theorem 6.1.1, Theorem 6.2.1). (1) Reflection formula: 
1-[ (0)= e()" 
OJ 
(2) Multiplication formula: For   A+ of degree d we bave 

1-I 1'(T) --- 

deg 
where Rd(Z) = 1-[deg a<d( z ÷ a monic 
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For a E K/A we denote by {a} the representative of a such that lal < 1, where 
I[ is the absolute value at oc = (-). For each a = y:. rai[a]  A, we define the 
F-monomial, e-mononfial, and r-mononfial, respectively, by 
r(a) = 5 -, r({a}) -', 
e(a)=  e(ai) ', and r(a)= (ai} '. 
a,0 a,0 
By abuse of notation, we also write F( n[ai, ., .]) to mean F( n[a]). This 
notation will also be applied to e- and r-monomials. In what follows a  K/A always 
means that a = {a} mfless otherwise stated. It is known that F(a) is algebraic over 
K if a =  m[a]  H(3, U). h, fact, we have shown that F(a) q- = -e(a) 
for some r  K*; see [BGY. Thm. 7.2]. Using the double complex, we can express 
r explicitly. Let Ç = ,+j=o Ç,J be a cycle in  such that Co,o = m[a, 1,0] 
and (ô+5)C = 0. Then (0+5)SC = (q- 1)Ç and (q- 1)Ço,o = 5SCo,o+0SC,-. 
Note that F((q - 1)Co,o) = F(a) q- and F(SSCo,o) = e(a)/r(a). We get 
r(0sC,,-)(a) 
(2.1) r(a)  =  . 
Now the following Kummer property of F-mouonfials, which originally is due to 
Sinha([Si]), is a direct result of the equality. X denote by N the cyclotomic 
function field of conductor . 
Theorem 7. Le a =  mi[ail  H2(J, U) and le  be the let common multiple 
of the denominators of a. Then N(F(a))/N is a Kummer extension. 
Proof. For any irreducible polynonfial p with degree d, we have 
= [b + u 
1. 
0]) 
i= 1 deg 

and thus 

() 
r(OS[b, p. -1]) = ((--1)dp)(-) - Rd("ï{b}) - . 
If 21q , then F(/)S[b, p,-1])  N. If 2 { q, since -r(--1)dp  Içp, we get the result 
by Eq (2.1). [] 
The key point in the study of F-mononfials by means of the double complex is 
that the factor F(OSC1,-1) in (2.1) is very simple, if C is a canonically liffed cycle, 
and for a general cycle C we get information about that factor using homological 
algebra. In fact, we have the following theorem and corollary, the lines of proof of 
which are again taken from [Da, Sect. 9]. 
Theorem 8. Let n be an even positive integer. Let C = ( Ci,-i be the semi- 
canonically lifted cycle from the basis class [0, ff, -ni, where fl is a square-free monic 
polynomial divisible by n irreducible polynomials. Let a =  m[a], where Co,o : 
, mi[ai, 1, 0]. Then F(a) - e KÇ. Furthermore, 
i) Iffl = pq with d = degp and e = degq. then 
F(a) -1 = e(a) lnod/'*. 
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ii) If n >_ 4, then 
F(OSC1,_I)  K* and F(a) q-1 = e(a) mod K*. 

Corollary. Let n >_ 4 be an even integer. Let a = -m[ai] represent the basis 
class [0, 9,-ni with 191 = n, hOt n«c«ssarily a s«mi-canonical r«pr«s«ntativ«. Thon 

F(a) q-1 _= e(a) mod/(*. 

The following result is an analogue of Theorem 11 in [Da]. 

Proposition 9. Let a =  mi[ail  H2(j,U). Then, with notation as in Proposi- 
tion 5, we have 
e(a) _0te(b)(q-1) ' 
for some b  ,4 and Or = -t-1. 

Pro@ Let C = ti+j=o Ci,d be a cycle iu NIK such that Co,o =  mi[ai, 1,0]. 
Then C - C t is a boundary. Let B = ti+j=l Bi,j be a chain in N]K such that 
(0 + 5)B = C - C . We have  
(,) = e(OB,o)e(SBo,1). Note that 
H deg --1 
«(o[o, p. o]) = p = (-) «-,  , 
deg u<deg p 
monic 

and 

e(0[a, p, 0]) = 1, if a # 0, 
and that e(6[a, 1.11) = -e(a) q-l. We get the result. 

Remark. As the refcree points out, if one defines 
sin a = q--Z-f, e({a}/sgn{a}) 
for a  K \ A, then, by making the obvious definition of sin a, one has 
sin a 
sin a  - (sin b) q-l, 
in strict analogy with Theorem 11 of [Da]. 
1 T-1 
Example. Let q 3, a [--4-7] - -T + 
= = [T(---(f-4], and t = 1. Then 
«(a) «(r) 1 
«(a')-- e(r)- T-, =-«(rT(T + I)) 2, 
since A = e(r  
T&+I)) satisfies the relation A 4 + (T + 1)A 2 + 1 = 0. 
If t = -1, then 
e(a__) =1 =«(0) 2. 

Thus 0 changes as t varies. 

Theorem 10. Let a = -mi[ai]  H2(j,U). Let f be the least common multiple 
of the denommators of the ai and let t  (Ail)*. Then 
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Pro@ With notation as in the proof of Proposition 9, let Bi,o =  gj [cj, pi, 0]. Ve 
may assume that B is a semi-canonieally lifted chain. Theu the denominators of cj 
divide [. From the proof of Proposition 9, it suffices to show that F(0B1,0) G Kf. 
It can be easily checked that 
Thus the result follows. 
r() 
Similarly, if a and a' represent the saine class lu H(d,ç), then 
3. CRITER[ON FOR H l(J,) 
The following conclusion is shown in [BGY, Cor. 4.2]. 
Lemma 11. Let b= m,[bd  H(d,ç). Then for aU 
Lemma 12. Let b = m,[b,]  H(d, ç), psenting any canonical basis class 
of Hl(, ) indezed by a monic square-free polynomial divisible by at least three 
pmes. Let Ç = +j=l Cci be a cycle such that C0,1 =  m[b, 1, 11. Assume 
that no terre of the form [0, p, 0] appears in C,o. Then 
 li  0 mod (q - 1). 
qag-  degp mod(q- 1) for b  A. Now 
Pro@ Note that IS(O[b,p,O]) - q_ 
follow the proof of Proposition 13 of [Da]. 
It is shown in [BGY] that b  H(d,ç) if and only if Ir(b)l = Ir(bt)l for 
any t  (A/I)*, where I is the least common multiple of the denominators of the bi. 
Here we give other proof of the necessity of this using the double eomplex. In 
this way one tan get SOlne more inforlnation about the e-monomials. 
Theorem 13. Let b =  m,[bd e Ni(d, ç) and et I be the east common multiple 
of the denominators of the bi. Then 
I(b)[ = Mb')l, 
for all t  (Ail)*. Fuhermore, we bave more information about e-monomials in 
the following special cases: 
First case: If b represents a canonical basis class of H 1 (, ), indezed by a single 
irreducible polynomial p. then e(b) (q-l) = e(b) (q-l) = p. 
Second case: Let b represent a canonical basis class of HI(, ) indezed by a 
monic square-free polynomial divisible by at least three primes. Let C =  Ci,_i+ 
be a cycle such that C0,1 =  mi[b, 1, 1]. Assume that no terre 4 the form [0. p, 0] 
appears in ÇLo. Then e(b t)  F for any t e (A/I)*. 
Pro@ The first statement follows by linearity from the two special ces, since 
Ir(b)l = le(b)I. Let C = C,,-,+I be a cycle such that C0,1 = mi[bi, 1, 1]. 
First Case: Let CL0 = [0, p, 0]. Then we know that e(-OC,o) = e(-OCl,o) = p. 
Thus 
However, e(6C0,1)= (-1)Gm'e(b) q-1 and e(6C,l) = (-1)Gm'e(bt)q-. Then the 
result follows from the fact that «te(b) = e(bt). 
Second Case: hl this case, e(OC,o) = 1. Then the result follows in the saine 
way as the first case, by using Lemma 11. 
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The following proposition is a direct consequence of Theorem 13. 
Proposition 14. Let b = 7, mi[bi] E HI(J,U). Assume that b represents a canon- 
ical basis class of HI( J, U), indexed by monic square-ffee polynomials divisible by at 
least three monic irreducibles. Let C be a cycle in ]K such that C = (i+j=o Ci,j, 
C0,0 = E -[b, 1, 01 ad (0 + )C = O. A.sume that no teTm 4 the f«m [0. p.-11 
appears in C,_. Then for each t coprime to tbe least common multiple of the 
denominators of the bi, we bave 
1-I sgn({¢t})' = 1-I sgn({b })'. 
Pro@ From Theorem 13, we know that e(b) E K*. Thus e(b t) = ge(b) = e(b) 
and, further, sgn(e(b)) = sgn(e(b')). Since ,ni = 0 mod (q- 1) froln Lemma 
12, we get the result. [] 

4. GALOIS PROPERTIES OF tÇ|(['(a))/tÇ 
Let a =  mi[ail  H2(j, IIJ), and let  be the least common multiple of the 
denominators of the ai. In this section we consider the extension ht(F(a)) over 
Let C = @ Ci,_i be a cycle in  su«h that C0.0 =  milan, 1, 0]. Write 
v = F(OSC,_) and F(aW  = ve(a)/r(a). 
Let a be an element of Gal(-/N) whose restriction to K t is «, where " is the 
separablc closure of K. Then 
F(a) q- = O,e(b)q_, 
where 0, and b are given in Proposition 9. Hence 
r(a) 
 K  0 = 1. 
aF(a) 
When is Kt(F(a))/K a Galois extension? The following theorems are the main 
results of the paper. 
Theorem 15. Assume q is odd. Let a = ,n[a]  H2(j,ç). and let [ be the 
least common multiple of the denominators of the ai. Then Kt(F(a)) is a Galois 
extension of K. 
In the number field case, D first shows the analogue of Theorem 8' [Da, Thm. 
15(C)] and then derives easily his main result [Da, Thm. 16]. In out case, Theorem 
8  is hard to prove because of the extra factor in the functional equation of the F- 
function. So we first show this theorem and then derive Theorem 8 . In the following 
proof we omit some complicated computations as suggested by the referee, because 
these are hot interesting and do hot offer any deeper insights. 

Pro@ The notation is as above. Using the analogue of [Da, Prop. 16] we can 
assume that the cycle C that a represents is semi-canonically lifted from a basis 
class. We know from Proposition 5 that e(a) and e(a ) lie in Kï, the maximal real 
subfield of K. Hence the signs of e(a) and e(a t) make sense. It is not hard to get 
sgn(e(a)/e(at)) = 1. 
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Let B be a semi-canonically lifted chain such that C - C t = (0 + ()B. Let 
Bo.1 =  nj[bj, 1, 1] and let b =  j[bj]. We see that sgn(e(b) q-l) = (_I)E, 
and thus 0t = (-1)E, by Proposition 9. 
Write Bl,o = _,lk[ck,Pk,O]. Using the facts that 0BLo + (Bo,1 = Co,o -C t 
0,0 
and IS(Co,o t 
- C,o) = 0, wc see that 
Z lk deg P k------- Z llj ulod 2. 
Since sgn(F(0[b, p, 0])q -1) = (- 1)aeg" we get 
sgn(F(OBl,O)q-1) = (_I)E aeg. = (_l)Enj = 
So we need to relate the sign of F(0B,o) q-) to v/rlV, for which we consider two 
cases separately. 
First Case: a represents the basis class [0, pq,-2]. Write d, = degp and 
Np = #{a- IllOlliC, da < dp, sgn({ t }) ¢ IFS2}. Using Theoreul 8 and the analogue 
of the classical GmlsS lelmna, we get by direct calculation 
V = (_l)d.N«+d«N  = sgn(F(Ol,0)q_l)" 

Thus we have the result in tlle fil'St case. 
Second Case: a represents a canolfical basis class indexed bv a squareffee 
polynonfial $ divisible by at least four distinct irreducibles. In this ce v  K by 
Theoreln 8, and so  = 1. We define SOllle operators on K as follows. 
I " [a, ,,k]  (a)[a, ,,k- l] forkodd, 
(a)--I 
J'[a,,,k]   [7-(")a,,,k- 11 for k even, 
/=0 
where ri(a) is defined by sgn(a) = 7 (") wit.h 0  n(a) < q - 1. Note that JI = O. 
Let C be the cycle obtained by the semi-canonical lifting of k. Our aim is to 
compute sgn(F(OB,o) q-l) = (_l)Edg,, where B1,0 = Elk[ck'Pk'0]" So we 
ollly need to consider the parities of the totM SUln of B {p } Let Ch -n - [0, , -ni. 
1,0  ,  
Then we bave 

B1,0 = (JOlt - JtlO)C2,_3 + Ji)IJOE + JOIOJF, 
for sonle chains E, F  ;]K. A straightforward but tedious colnputation shows that 
TS(B.2) = 0 mod (q - 1). Then sgn(F(OBl,O) q-l) = 1, which implies the result. 
Lenlma 12 and Proposition 14 are used in the course of the computation. [] 

l, ç know froin Proposition 9 that FKI(-é) is a Galois extension of K, 
where IF is the quadratic extension of ]Fq. But in the second case we get more. 

Theorem 16. Let a  H2(J,U) represent a canonical basis class indexed by 
a squarefree polynomial  divisible by al least four distinct irreducibles. Tben 
KI(-) is Galois over K. 

e(a) e(a) __ Ote(b)q_l for some b, KI(-a) is Galois over K 
Pro@ Since «,() - (,) - , 
if and only if 0 = 1. Thus the result follows from the proof of Theorem 15. [] 
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We have seen in the proof of Theorem 15 that if a represents a canonica.1 basis 
class indexed by a square-free polynonfial $ divisible by at least four irreducibles, 
then the total degree of B {pI is divisible by q- 1 for any prime p and 0t 1. Hence 
F(0B1,0)  K*. Fron the proof of Proposition 9, we have 
( @a ) q-1 = F(OB,o)q-IF(Bo,) q-' = F(OB,,o)q-'r(Bo.1)l-qot ). 
Finally, we have the following stronger version of Theorem 8. 
Theorem 8q Let n R 4 be an even positive integer. Let C =  Ci,- be the 
semi-canonicall lifled ccle from the basis class [0. ,-ni, where  is a square-free 
monic polnomial dwisible b n irreducible polnomials. Let a =  mi[ai], where 
Co,o =  mi[ai, 1,0]. Thon 
r(a) q = re(a) ad r(a) q = rsqe(a ) 
for some r, s  K*. 
In [BY] we proved that if a e H2(J, U) represents the bis cls [0, çq.-2], then 
Kq(-) is nonabelian over K. Here we also give an example where K(F(a)) 
is not abelian over N. 

Example. We a£Sllme that q = 3. \Ve can easily compute that the cycle C = 
Co,o @ C- @ C2,-2 is the senfi-canonically lifted cycle of the canonical basis class 
[0, T(T + 1),-2], where 
C,_ = [O,T(T + 1),-21, 
C,,-1=[ T+I-1]- [----,T,-I], 
' ' +1 
[--{---1 0]-[TT-+ll)1,0]. 
Co,o = + 1' " 

Thus 

1 T-1 
aT(T+U = [--1- [T(T + 1) ]" 

A simple computation gives 
F(ar(T+I))  = V/ T e(T--@i-) 
T + 1 e( (T--1),rr  = 
T(T+I)! 
using the relation F(aT(T+I)) 2 -- F(SCo,o)F(0SC1,_I). 
a-T+. Let A = e( 
T(+))" Then we tan check that 
u _ A2 u A2 ' u 
-- -- -- = and -- 1. 
O'ît Tît O'Tît 
Let 
Va = 

Let a = OET-I alld T = 

and vor = l. 
Let   Gal(/Q(F(a))/K) be the lifting of r/ Gal(/q/K) such that vnv = v/-. 
Then using the fact that A 4 + (T + 1)A 2 + 1 = 0, we get ¢ = --à on v/-. 
Remark. It would be very interesting to know whether Kf(F(a)) is abelian over 
K, or equivalently by the last theorem, whether Kt(--ê) is abelian over K, 
if a represents a canonical basis class indexed by a monic square-free polynomial 
divisible kv at least four irreducibles. In the classical case this is verified by Das 
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([Da], Theorem 21), with the aid of a theorem of Deligne (Theorem 7.1S(b) of [De], 
Theorem 19 of [Da]), that is, 
(4.1) (F(a)  F(a t) 
a tF(a) ] -- F-i-) ' 
where a E H2(J,U), a,   Gal(R/K) and the restriction of a on N I is af. Here 
r(a) 
/" is the separable closure of N. Note that it is easy to see that Tr(a)  Nf by 
Theorem 8. We also note that when a represents a semi-canonically lifted cycle 
from a basis class, then 
(F(a') q-1 Ç F(a') ) q- 
« \r(a) = r(a') ' 

using Theorem 8'. 

If one disposes of an analogue of Deligne's theorem above, then one can easily 
show, with the aid of Theorem 8' and following the saine method as in [Dal, that 
Kf(F(a)) is abelian over K if a satisfies the above conditions. 
Sinha [Si] has proven Deligne's reciprocity for flmction fields (Theorem 7.i8(a) 
of [De]) using Anderson's theory of solitons. Thus one ma also use the theory of 
solitons to prove the analogue of Deligne's theoreln (Theorem 7.18(b) of [De]), which 
is beyond the reach of the present paper. However, we hope that an eleluentary 
proof using the double complex ma.v be possible. Anderson's recent work on the 
epsilon extension yields an elementary method to show that K(q--) is abelian 
over K. 
Let K ab be the maximal abelian extension of K and let G ab = GaI(K«b/K). 
Following Anderson [An2], two of us [BY] defined an injective homolnorphism 
2 
D: HO(cab, Is'ab*/Ix "ab*(q-1)) - A Hl(Cab'Z/(q-- 1)) 
and showed it is an isomorphism. We can also express D explicitly; see [BY, Sect. 
3.5] for detail. The map D has the property that for u  Iç b*, 
D(u mod Iç b*(q-)) = 0 = -/-  Iç b. 
This proposes an elenmntary method for showing that q-éa)  K b if a satisfies 
the condition above. But the calculation of D(-/ê-) would be too complicated 
to take. About this question, we refer the reader to Renmrk 4.4.2 in [An2]. 
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REMARKS ABOUT UNIFORM BOUNDEDNESS 
OF RATIONAL POINTS OVER FUNCTION FIELDS 

LUCIA CAPORASO 

ABSTRACT. We prove certain uniform versions of the Mordell Conjecture and 
of the Shafarevich Cnjecture for curves over function fields and their rational 
points. 

1. INTRODUCTION AND PRELIMINARIES 

A curve X of genus at least 2 defined over a flmction field L has only finitely 
many L rational points, unless it is isotrivial. Similarly, a curve of genus at least 2 
defined over a number field F has a finite set of F-rational points. These well-known 
facts are celebrated theorems of Y. Manin and G. Faltings, originally conjectured 
by L. J. Mordell and S. Lang. 
We study here questions of uuiformity for the cardinality of such sers of rational 
points, in the function field case. For number fields, there are a number of open 
conjectures, such as the following (Uniforin Mordell Conjecture for immber fields): 
Fix g >_ 2 and a number field F; there exists a number B9(F ) such that any curve 
of genus g defined over F bas af most Bg( F) rational points over F. Interest in 
such problems was revived after it was proved in [CM] that the conjecture above 
is a consequence of a famous, open, conjecture (usually attributed to S. Lang and 
E. Bombieri) on the non-density of rational points in varieties of general type (sec 
also [Ab], [AV] and [Pal). 
In this paper we investigate similar issues for curves over flmction fields. Some 
partial reslts were obtained in [Mi] and in [C] were the existence of uniform bounds 
for the sers of rational poiuts is established. Such bounds depend on suitable 
numerical invariants of the function field, on the genus g of the curves and on the 
degree of the locus of bad reduction (that is, the locus of singular fibers). 
We shall also study here the strictly related "uniform Shafarevich problem"; a 
famous theorem of A. N. Parshin and S. Ju. Arakelov ([Ar] and [P]) states that if B 
is a smooth complex curve and S C B a finite subset, then there exists only a finite 
number of non-isotrivial families of smooth curves of fixed genus g >_ 2 over B - S. 
Parshin first proved it under the assumption that S = ; Arakelov generalized it a 
few years later. In [PI Parshin shows also that the above theorem implies finiteness 
of rational points for non-isotrivial curves of genus at least 2, providing the above 
mentioned link between the Shafarevich problem and the Mordell problem, Recall 
that his argument, known as the "'Parshin trick", is valid for both number fields 
and function fields. 
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A first uniform version of the theoreln of Parshin and Arakelov above is obtained 
in ICI. We here generalize it by a stronger uniform result valid for familles of curves 
over bases of any dimension. This is done in Section 2, where we obtain bounds 
(for the sets of curves with fixed degeneracy locus as well as for the sets of rational 
points) that only depend on the degree of a polarization on the base variety, and 
on the degree of the locus of bad reduction. A stronger result can be obtained 
for curves having good reduction in codimension 1 (Theorem 3). In Section 3 we 
will consider families with maxilnal variation of lnoduli, using the geometry of the 
moduli space of curves to approach our problcms. 
}Ve work over C; by t r we shall dcnote a smooth, irreducible, projective variety 
over C, whose ficld of rational functions will be L :----- C(V). Special interest will be 
given to varieties of dimension l, for which we shall use the following notation: B 
is a SlUooth irreducible curve and K its field of rational functions. ,Ve fix integers 
q >_ 0, g _> 2 and s >_ 0 throughout. The genus of B will be denoted by q. 
We shall consider smooth curves of genus g ovcr the function field L (or K), which 
can also be viewed as familles of curves over I', such that there is a nonempty open 
subset of V over which the fibers are all smooth. We shall alwavs assume that such 
a family (or curve) is hot isotrivial, i.e., the smooth fibers are hot all isomorphic. 
To be more precise, we introduce the following sets: let B be a fixed curve and 
let S C B be a finite set of points. 
Definition. Fg(B, S) shall denote thc set of equivalence classes of non-isotrivial 
families f  X - B such that X is a smooth relatively minimal surface and 
the fiber Xb over every b ¢ S is a smooth curve of genus g. Two such families 
ri.  Xi - B for i = 1, 2 are equivalent if there is a conmmtative diagram 
X1 -- X2 
B 
- B 
where the wo horizontal arrows are biratiolml lnaps. 
Using a different termiuology, Fg(B, S) is the set of K-isomorphism classes of 
non-isotrivial curves of genus 9 over K, having good reduction outside of S. The 
theorem of Parshin and Arakelov says that F(B, S) is finite. Theorem 3.1 of [CI 
states that there exists a number P(g,q,s) such that ]Fe(B, S)I <_ P(g,q,s) for 
every curve B of genus q and for every subset S having at most s points. V'e show 
here (in the end of Section 2) that this result is sharp in the sense that such a 
bound must depend on s. 
We are interested in function fields of higher transcendence degree. We can 
generalize the definition of Fg(B, S) as follows. Let T C I" be a closed subscheme. 
Definition. Fg(V, T) shall be the set of equivalence classes of non-isotrivial families 
of slnooth curves of genus g over V - T (the equivalence relation is the same as 
above, with B replaced by V -T). 
By the existence and unicity of lninimal models for smooth surfaces, this defini- 
tion coincides with the previous one if dira V = 1. It follows from the results in ICI 
(3.4) that Fg(V,T ) is finite. Our best result on Fg(V,T ) is Theorem 1. 
If X is a curve defined over a field L, we shall denote by X(L) the set of its 
L-rational points. If X has genus at least 2 and it is not isotrivial, the theorem 
of Manin says that X(L) is finite. Consider now the Uniformity Conjecture for 
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rational points over function fields, which can be stated as its arithmetic analogue: 
Let L be a function field over C and let g > 2 be an integer. There ex,sts a number 
Ng(L) such that for every non-isotrivial curve X of genus g defined over L we 
bave ]X(L)I <_ Ng(L). For ïesults ïelating it to the Lang Conjectures about the 
distïibutions of ïational points on vaïieties of geneïal type, see the woïk of D. 
Abïamovich and J. F. Voloch [AV]. 
Such a conjecture ïenmins open; ouï ïesults in that direction are Theoïems 2 
and 3 and Proposition 4. 
A final piece of notation, llIg denotes the moduli vaïiety of smooth cuïves of 
genus g and llg its compactification via Dcligne-Mumfoïd stable cuïves. They are 
both integïal, normal vaïieties of dimension 3g - 3. A univeïsal curve exists only on 
a proper open subset of llg (and of llg). In particular, a morphisnl op" Z  llg 
does uot necessm'ily corne from a family of curves over Z. If this is the case, that is, 
if theïe exists a fmnily of slnooth cuïves X  Z such that for eveïy z C Z, çb(z) 
is the isomoïphism class of the fibeï of X over z, we shall sa.v that çb is a moduli 
map. 

2. UNIFORMITY RESULTS FOR FUNCTION FIELDS 
OF HIGH TRANSCENDENCE DEGREE 
We start by a uniform generalization of the theorem of Parshin and Arakelov. 
The result below is a strenghtening of 3.4 and 3.5 in IC; in fact, the bound H here 
is independent of the dimension of V and of r. Such an improvement is obtained 
by a small technical modification of the lnethods in [C. 
Notice that the statement below ïemains tïue if l" is ïeplaced by an integïal, 

possibly singular, projective variety. 
Theorem 1. Let g >_ 2. d >_ 1. s 

The proof is essentially the saine. 
>_ 0 be ,ed integers. There exists a number 

H(g, d, s) such that for any smooth, irreducible variety l" C  of degree d. for any 
closed subscheme T c V of degree s, we bave IFg(V,T)I < H(g,d,s). Moreover, if 
T bas codimension af least 2 in V, then the bound H does hot depend on s. 
Proof. Step 1: Slicing V into curves of bounded genus. Considering one-dimensional 
hypeïplane sections of V, we see that V can be coveïed by smooth cuïves of degïee 
d passing thïough any of its points; it is a well-known fact that the genus of a curve 
of degïee d in projective space is at most equal to ( d - 1 
2 ): just pïoject the cuïve 
biïationally onto a cuïve of degïee d in F 2. Let 
q=q(d)=( d-1 )2 
so that V is covered by curves of geometric genus at most q. 
Step 2: Unzform boundedness of moduli maps. By Theorem 3.1 in [C], for any 
fixed 9, q', and s' there exists a number P(9, q', s') such that for anv smooth curve 
B of genus q', for any subset S c B ofat most s' points, we have that [Fg(B, S)[ <_ 
P(g, q', s'). 
Define 
H' = max P(9, q,s') 
q _q,s _s 
so that H' only depends on g, d, s; let U = V - T. 
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We claire that U has at most H' moduli maps to AIg; that is, we claire that 
there exist at most H' non-constant, (regular) morphisms ¢  U  Mg such that 
there exists a (hot neeessarily unique, sec below) family of smooth cm'ves over U 
whose moduli map is ¢. By contradiction, let n > H' and let us sume that there 
exist ¢ ...... ¢n distinct such moduli maps ¢i " U  Mg. Let Xi  U be a 
non-isotrivial family of smooth curves corresponding to ¢i (since ¢i is a moduli 
map, such a family exists, but it is hot necessarily unique). Let U' C U be the 
nonempty open subsct where ¢i() ¢ ¢j(u) for every  E U' and for every pair of 
distinct i, j. Let p  U' and let Fi = ¢¢i(p); since 
through p is a proper closed subset of U'. Therefore, there exists a curve B C 
of gems at q'  q such that p  B and such that B 
thus the restriction of Xi to B is hot isotrivial h»r every i. Let S = (BT)ed. Let 
}  B be thc smooth relatively minimal completion over B of the restriction of 
.k' to B. By construction, ,. .... Yn are different elements of Fg(B, S), which is 
a contradiction, since (B, S) hem at most P(g,q', s')  H' < n elements. This 
proves the claire. Notice that if T h codimension at least 2 in I" we can always 
choose our B so that it does not interscct T at ail, and hence S can be taken to be 
the cmpty set and H' does hot deI»end on s. 
Conclusion. Given a moduli map ç  [r 
 moduli Inap is uniformlv boundvd: in fact, it is bomded above by a flmction of 
g only (see [C], Lemma 3.3); hence we are done. 
A similar argument yields the following unifonnity statement for rational points, 
stronger than 4.3 and 4.4 in ICI: 
Theorem 2. Let g  2. d  1, s  0 be flxed integers. There exsts a number 
N(g, d, s) such that for any smooth, irreducible variety V C F  of degree d, for 
any closed subscheme T C I" of degree s and for any non-isotvial curve X of 
genus g deflned over L = CV) and having good reduction outside of T, we bave 
]X(L)]  N(g, d, s). Moreover, ifT has codimension af least 2 in I, then the bound 
N does hot &pend on s. 
Pro@ Step 1: Repeat word for word Step 1 in the proof of the previous theoreIn. 
Step 2: Theorem 4.2 in ICI skvs that if g, q', and s' are fixed nonnegative integers, 
there exists a Immber M(g, q', s') such that for any curve B of genus q', for any 
subset S of at most s' points in B, and for any curve XB  Fg(B, S) we have that 
Ix(c(B))l  M(g,v', '). 

Arguing as in the proof of 4.4 
N(g, 
will suffice for our statement. 

of [C] one gets that defining 
d,s) := max M(g,q',s') 
q'<__q,s<__s 

To conclude, we show that for curves having good reduction in codimension 1, 
stronger finiteness results hold. Let L be a flmction field over C and let r- be a 
smooth, projective, complex variety of positive dimension such that L = C(I'). 
Definition. Let C (L) be the set of L-isonorphism classes of non-isotrivial curves 
of genus g over L having good reduction in codimension 1. 
In other words, C29(L) is the set of equivalence classes of non-isotrivial familles 
X --+ V of curves of genus 9 over V such that there exists a closed subscheme 
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T C V of codimension at least 2 with the property that Xv is Slnooth for every 
v_T. 
Theorem 3. 
a) C (L)  fii. 
b) There exists a numl)er N2(L) such tbat for every curve X e C(L) we bave 
IX(L)I N N(L). 
Pro@ We shall use moduli maps. Dcnote by I(L) the set of equivalence classes 
o non-constant rational lnaps ¢ I"  Ia such that there exists an open subset 
U ç C 1  with the foltowing properties: 
1. The complement of U ç has codimension at ledit 2 in 
2. ¢is regular on U ç. 
3. There exists a (non-isotriviat) family of smooth curves of genus g over U 
such that ¢is its moduli map. 
4. Two such nmps ¢and ¢ are equivalent iff they coinçide ou some (nonempty) 
open subset of V. 
Therc is a natural surjcctive lnap of scts: 
sending a curve over L to its moduli map (it is easy to see that p is well defined). 
Now, p h finite fibers (Lemma 3.3 in ICI) and is surjective by definition. Thus 
C(L) is finite if and only if I(L) is finite. 
Part b) is an immediate consequence of part a), by the theorem of Manin. 
will prove our result by showing that M(L) is finite by induction on dira V. 
i  = 1, t. t¢ it « C(L) « « (L) i t th « Phi 
(the tocus of bad reduction being empty in such a case). Then let diln V  2 and 
suppose that 5I(L) is infinite. Notice that 5I(L) is dominated by a union of 
finite sets  follows: if T is a closed subset of 1 , dcnote by 5I(1 T) the set of 
equivalence cluses of moduli maps to 5I that are regular on V- T; then Ma(V. T) 
is finite, by Theorem 1 and Lenmm 3.3 in ICI. ç bave a natural, surjective map 
 (:T)  I(); 
codimv T2 
hence, if M(L) is infinite, so is the union on the le,-hand side. Then there exists 
a countable collection (T, n  Z}, with Tn a closed subset of V of codimension at 
least 2, such that the set 
nZ 
is infinite. Now, 5I itself being a countable set, we shall put an ordering on it: 
M = {¢, i  }. 
For every pair of distinct i,j, denote by U i'j the nonempty open subset of 1" such 
that U i'j C U ç'U ç and Ci(u) ¢ OE(u) for every u e U i'j. The Ui'Ys form a 
countable collection of nonempty open subsets of V, whose intersection I is dense 
in V. Let p  I and, for every i  N, let F = (¢)-¢(p) be the fiber of ¢ through 
p. Since ¢ is non-constant (by assumption), Fi is a proper closed subset of V. 
Thus, the complement of ie Fi intersects I in a subset J, with J dense in V. 
Fix a (non-degenerate) projective model of V in some projective space. Then there 
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exists a hyperplane H such that p E H, such that H Cl J -¢ 0 and such that H does 
hot contain any T. Letting W = H cl V, we can firtherlnore choose H so that 
is smooth. By construction we have 

(a) dimW=dimV-1; 
(b) dira T  W = dira T - 1 _< dim W - 2 
(since H does not contain any T); 
(c) Ve i E M, the restriction ¢iw is not constant 
(since p  IV and W J ¢ 0); 
(since IV  I ¢ 

hence çblw e M2(C(|V))and the restriction to IV gives an inclusion (by (d) above) 
.AI , M2(C(I¥)). Thus M2(C(IV)) is infinite. This is a contradiction with the 
iuductive aSsulnption. [] 

Sec [Nid] for ait analogue over Q. Part a) of this result should be compared with 
the examples of A. Beauville (in lB], section 5) or with the example below. They 
show that the assumption that the curves have good reduction in codimension 1 is 
crucial; that is, a) is false without that assumption. On a different vein, compare 
also with Proposition 4. The example that we are going to describe shows that there 
is no hope of getting a substantially stronger uniform version of the Shafarevich 
Conjecture for function fields; in other words, any uniform bound on ]Fe(B,S)] 
must depend on the degree of S. 
What happens to the cardinality of Fg(B, S) when s grows while g and q (or 
even B) stay fixed? The way we defined F(B, S), it is an exercise to show that 
its cardinality is not bounded; but this is just because the families parametrized by 
Fg(B, S) are not required to have a singular fibers over S. The interesting question 
is about the asymptotics of the cardinality of that subset of F (B, S) parametrizing 
families of curves that have a singular fiber over every point of S. We will make 
this precise now, describing an example suggested by J. de Jong, showing that the 
set of fibrations with fixed degeneracy locus is hot bounded, as the cardinality of 
the degeneracy locus grows. 
Fix g > 2 and B = IP . Given a subset S c IP  denote by F(S) c F(IP , S) 
the set of all genus g non-isotrivial fibrations X  IP  sueh that the fiber Xb is 
smooth if and only if b  S. 
Let S = {a, .... a} be a set of generic points in IP , and let I U J = {1, 2 ..... , s} 
be a partition of {1, 2, ...., s} in two disjoint subsets such that ]I I = 5. Define a 
non-isotrivial fibration X of curves of genus 2 over IP  by the affine equation 

yi = (x -- t)IIieI(X " ai)IIej(t - a) 

with t an affine coordinate in IP . For t  S (and t -¢ oe ) we get a smooth curve of 
genus 2. For t = ai with i  I, we get a nodal curve and for t = a, j  J, we get 
a singular, non-reduced curve. Thus XI  F(S U oe) and, by varying the partition 
8 " 
I U J, we get a total of ( 5 ) different such fibrations. Hence the cardinality of 
F(S) goes to infinity, as ISI grows. 
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One final word about this example. 
First, we make two conmlents: the given family has fibers of genus 2, but of 
course one can construct the saine example for any genus (just replace the integer 
5 by a higher odd number), obtaining families of hyperelliptic curves. 
The second comment is about those singular fibers over ai with j E J that 
are hot stable curves; their semistable reduction is actually a smooth curve. The 
remaining 5 fibers over ai are instead nodal. In other words, the nloduli map ¢I 
associated to the family X1  1, 
(such that ç(t) is the isomorphism class of the fiber of X over t) intersects the 
boundary &2 in exactly 5 points, regardless of the cardinality of S. 
We ask: 
(a) Can one find similar examples whose fibers do uot beloug to any proper 
closed subset of M? 
(b) Is the saine "unboundeduess" result true for falnilies of stable curves? In 
other words, does there exist a similar example all of whose singul fibers 
are nodal? 

3. UNIFORMITY FOR "TRULY VARYING" CURVES 
This section contains results that are indepeudent of the degeneracy locus. Given 
a family X  V of generically smooth curves of genus g over I , we get a natural 
rational map ¢ : V  /I 9 (regular on a nonempty open subset of V). The 
dimension of the image of ¢ is called the variation of moduli of the family; we shall 
say that the family has maximal variation of moduli if 
dira Ira ¢ = rein{dira V, 3g - 3}. 
We shall say that a curve over L = C(V) has maximal variation of moduli if a 
coïresponding family of curves over V does. 
Thus the condition of having maximal variation of Inoduli can be interpreted 
as saying that the family (or the curve) is truly varymg and can be viewed as a 
generalization of the non-isotriviality condition. Obviously, if the base field has 
transcendence degree 1, a curve is non-isotrivial if and ouly if it has maxinal vari- 
ation of moduli. 
Definition. Let L be a function field. We defiue Cg(L ) to be the set of L-isomor- 
phism classes of curves of geuus g defined over L and having maximal variation of 
moduli. 
Proposition 4. Let g >_ 24 and let L be a functon field of transcendence degree 
3g - 3. Then 
a) C(Z) i 1nite. 
b) There exists a number N(L, g) such that for every curve X of genus g 
defined over L and having maximal variation of modulL we have IX(L)I <_ 
N(L,g). 
c) There exists a function Pg(n, m) such that for every V of general type, we 
have ICg(L)I <_ Pg(dim V, Kdv V). 
Proof. The assumption g >_ 24 implies that Mg is of general type (for this famous 
result of J. Harris and D. Mumford we refer to [HMu] and to 6F in [HM]). 
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Denote by R(V, Ma) the set of dominant, rational nlaps flore V to M a. A 
theorem of Kobayashi and Ochiai [KO] implies that, fil a being of general type, 
R(V,/-a) is finite. Notice now that there is a natural hijection between Ca(L) and 
R(V,/la): to a truly varying curve X of genus g over L we can assoçiate its moduli 
map ex E R(V, ll-ïa). The fact that such a correspondence is bijective follows from 
the existence of the universal curve over an open subset of M a. Thus Ca(L ) is 
finite. 
By the thcorem of hlanin, any curve in Ca(L ) bas a finitc set of L-rational points. 
Thus part. b) follows immediatcly from a). 
Part c) is proved like part. a); we can in this case apply a strcngthening of the 
theorem of Kobayashi and Ochiai providcd by T. Bandman and D. Markushevich. 
From [BM] we obtain that, V and M a beiug of general type and/I a having canon- 
ical singularities (Theorem 1 in [HMu]), there exists a fimction of g, of dira I" and 
of Kî) m v bounding t he cardilmlity of R(V,/l a) and hence t hat of C a (L). [] 

Let u : C a ----,/I be the uuiversal curve over the moduli space of automorphism 
free smooth curves of genus g, so that the fiber of u over the point corresponding 
to the curve X is X itself. 
It is a well-known fact (see [HM], 2D) that u has no rational sections; thus, C a 
has no rational point over the fimction field of/119. In fact, much more is known: 
the Picard group of C a is generated over the Picard group of M a by the relative 
dualizing sheaf ¢zu; therefore, a multisection of u nmst have degree over M equal 
to a nmltiple of 2g - 2. 
We apply this to obtain that if V is a variety of dimension 3g-3 and X is a curve 
of genus g over L having maxinml variation of moduli, then a necessary condition 
for X t.o bave a rational point over L is that it.s moduli map bave degree equal to 
a multiple of 2g - 2. This follows easily by looking at the commutative diagram 

X  C a 
,ITo o 1  
V - M a 

where the horizontal arrows are rational tnaps and  is the rational section cor- 
responding to arational point, of X over L. Let.  = "7°  i   C a and let. 
p  hn   Ma; by what we said, degp = n(2g- 2) for some integer n. \Ve finally 
obtain 

deg çbx = deg "r-deg p = m(29 - 2), 

where by deg ex we mean the degree of the restriction of ex" to the nonempty open 
suboet of V and where ex is a regular and finite luap. Let us call such a number 
deg ex the modular degree of a family X  V; this definition is general, provided 
that X - V has maximal variation of moduli and that dira I" _< 3g - 3. We just 
proved the following 

Lemma 5. Let V be a variety of dimension 39 - 3 with function field L and let 
X be a smooth curve of genus g over L having maximal variation of moduli. Then 
either X(L) =  or the modular degree of X is a multiple of 2g - 2. 
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The following well-known conjecture is open: 
Geometric Lang Conjecture. Let Il be a variety of general type defined over 
C_.. Then there exists a proper closed subvariety Zw of IV containing all positive- 
dimensional subvarieties of II" that are not of general type. 
In particular, according to such a conjecture, ail curves in II having genus at 
most 1 are contained in Zu-. 
Consider now Mg, and let Zg C 11Ig be defined as the closure of the union of 
ail integral curves in lIg having geometric gems at most equal to 1. Since lIg is 
of general type if g _> 24, the above conjecture would imply that Zg is a proper, 
closed subset of 11Ig for ail g _> 24. 
As a consequence, we get the following. 
Lemma 6. Let g > 24 and let B be a curve of genus q. The Geometric Lag 
Conjecture implies tlat if X ----+ B is a on-isotrivial family of curves of genus g. 
passing through the geeral point of 11Ig, tlen tle modular degree of X is at most 
q-l. 
Pro@ As we mentioned above, the union of all curves in 11Ig of genus at most 1 is 
contained in a proper closed subset Zg of/lIg. The condition that the given family 
of curves goes through the general point of kIg, combined with the Geometric Lang 
Conjecture, implies that hn bx ç Zg. Thus the geometric genus of Ira çx is at least 
2. Bv the Fliemann-Hurwitz formula, the degree d of a dominant nmp of a curve B 
of genus q onto a curve C of geometric genus p > 2 is at most equal to q - 1. In 
fact, the formula gives 
d_2q-2-r q-1 
<--<q-1 
2p-2 - p-1 - 
since r _> 0 (being the degree of the ramification divisor) and p > 2 by assumption. 
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IRREDUCIBILITY OF EQUISINGULAR FAMILIES OF CURVES 

THOMAS KEILEN 

ABSTRACT. In 1985 Joe Harris proved the long-standing claim of Severi that 
equisingular families of plane nodal curves are irreducible whenever they are 
nonempty. For families with more complicated singularities this is no longer 
truc. Given a divisor D on a smooth projective surface  it thus makes sense 
to look for conditions which ensure that the family l/i(çl,... ,çr) of irre- 
ducible curves in the linear system IDll with precisely r singular points of 
types çl,...,q is irreducible. Considering different surfaces, including gen- 
eral surfaces in PC and products of curves, we produce a sufiïcient condition 
of the type 
 aeg (x(s,)) 2 < . (D - 
Kr.) 2 , 
where q is some constant and X(ç,) some zero-dimensiona[ scheme associated 
to the singu]arity type. Out results carry the same asymptotics as the best 
known results in this direction in the plane case, even though the coeflïcient 
is worse. For most of the surfaces considered these are the only known results 
in that direction. 

1. INTRODUCTION 

Equisingular families of curves have been studied quite intensively since the 
last century. If we fix a linear system IDIt on a smooth projective surface E and 
singularity types $1, .... ,S, then we denote by V irr VlD II/irr[cçt,..., S) the variety 
of irreducible curves in ID]t with precisely r singular points of the given types. 
The main questions are whether the equisingular family V r is nonempty, smooth 
of the expected dimension, and irreducible. For results in the plane case we refer 
to [GLS98c], [GLS00], and results on the first and the second question on other 
surfaces may De round in [GLS97], [GLS98a], [CC99]. [Fla01], [Che01], [KT02]. 
In this paper, for the first time, the question of the irreducibility of V i for a 
wider range of surfaces is studied. As already families of cuspidal curves in the 
plane (cf. [Zar35]) or nodal curves on surfaces in 1?: (cf. [CC99]) show, in general 
we cannot expect a colnplete answer as for families of plane nodal curves, saying 
that the family is irreducible whenever it is nonempty. All we may hope for are 
numerical conditions depending on invariants of the singularity types, the surface 
and the linear system, which ensure the irreducibility of V r. 
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The main conditiou which we get (cf. Section 2) looks like 

r 
(1.1) E deg (X(S,)) 2 

< 3" (D - Kz) 2, 

where 3' is SOllle constant. Applying the estimates (1.6) for deg (X($i)) from Sub- 
section 1.3 we could replace (1.1) by 

(1.)  (s,) 2 < . (D - n)  

in the case of almlytical types, and in the topological case by 

(1.3) 

 (,(s) + ) < . (D - K) . 

In this section we introduce the basic concepts and notation used throughout 
the paper, an(1 we stale sevcral imp()rtant knowll facts. Section 2 contains the 
maiu results and thcir proofs, olnitting the technical dctails. These are presented 
in Section 3 and Section 4. 

1.1. General Assumptions and Notation. Throughout this article }2 will de- 
note a smooth projective surface over C; N dcnotes the set of nonnegative integers. 
W'e will denote by Div(}2) the group of divisors on }2 and by Kr its canonical 
divisor. If D is anv divisor ou }2, Or (D) shall be the corresponding invertible sheaf, 
and we will sometilnes write H"(X, D) instead of H'(X, Ox(D)). A curve C C E 
will be ail effective (nonzero) divisor, that is, a one-dimensional locally principal 
scheme, hot necessarily reduced: however, ai1 irreducible curve shall be reduced bv 
definition. IDI denotes the system of curves linearly equivalent to D. We will 
use the notation Pic(}2) for the Picard group of }2, that is, Div(}2) modulo linear 
equivalence (denoted by ,,«), and NS(}2) for the Néron-Severi group, that is, Div(}2) 
modulo algebraic equivalence (denoted by ,). Given a reduced curve C C E, we 
will write 9(C for its geometric 9enus. 
Given any closed subscheme X of a schelne Y, we denote by ,Tx = ,Tx/y the 
ideal sheaf of X in Oy. If X is zero-dimensional, we denote by #X the llulnber 
of points in its support supp(X) and by deg(X) = e dimc(Oy, z/,.\-/y.z) its 
degree. 
If X C }2 is a zero-dimensional scheme on }2 and D  Div(}2), we denote by 
],.%c/r(D)[ the linear system of curves C in [Dit with X C C. 
If L C }2 is any reduced curve and X C }2 a zero-dimensional scheme, we define 
the residue scheme X : L C }2 of X by the idem sheaf OE\':L/r = ,.Tx/z : ,.TL/r with 
stalks 

,ff'X : L I E , z : -."(, I E , z " - L I Z , z , 

where ":" denotes the ideal quotient. This leads to the definition of the trace scheme 
X ç L c L of X via the ideal sheaf ,TXaL/L given by the exact sequence 

o -- Jx:r/z(-L)  Jx/z ----> .7»:r/r ----> o. 
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1.2. Singularity Types. The gerln (C,z) C (E,z) of a reduced curve C c E 
at a point z Ç E is called a plane curve singularity, and two plane curve singu- 
larities (C,z) and (C',z') are said to be topologically (respectively, analytically 
equivalent) if there is a homeolnorphisln (respectively, an analytical isonmrphism) 
d2  (E, z) --, (, z') such that (I)(C) = C'. We call an equivalence class with 
respect to these equivalence l'elations a topological (respcctively, analytical) sin- 
gularity type. The following are known to be invarialtS of the topological type 
S of the plane curve singularity (C, z): r(S) = r(C,z), the nulnber of branches 
of (C,z); res(s) = res(C,z), the codilnelsion of the /t-constant stratuln in the 
selniulfiversal deforlnation of (C,z); a(ç) = a(C,z) = dilnc (oe.(gd,,z/(gc,z),. the 
delta invariant of S, where ," (C,z) - (C,z) is a normalisation of (C,z); and 
p(S) p(C,z) = dilncOz,z/( OI of 
= -, -), thc Milnor number of S, where f E (9,z 
denotes a local equation of (C, z) with respect to the local coordilmtes x and y. For 
the analytical type S of (C, z) we have as additional invariant, the Tyurina number 
oI 
of S, defined as r(S) = r(C,z) = dilncOr,/(f, o°--, ). We ,'ecall the relation 
2((ç) = p(ç)+r(ç)-1 (cf. [Mil68, Chaptcr 10]). Furthermore, since the &constant 
stratum of the senfiUlfiversal dcforlnation of (C, z) contains the p-constant stratuln 
and since its codimension is just a(S), we have ((S) _< res(S) (see also [DH88]); 
hence 

(1.4) 

.(s) _< 2(s) < 2«(s). 

1.3. Singularity Schemes. For a reduced curve C C E we recall the definition 
of the zero-dilnensional schelnes Xe(C) C X(C) and Xe(C) Ç X«(C) from 
[GLS00]. They are defined by the ideal sheaves 
and x-(c)/, respectively, given by the following stalks: 
 x«»(c)/,z = I¢(C, z) = {g e O,z  f + g is equisingular over C[e]/(e2)}, 
where f Ç OE,z is a local equation of C at z. I«s(C, z) is called the equisingu- 
larity ideal of (C, z). 
{g  O,z [ g goes through the cluster C« (C,T'(C,z))}, where 
Jx«c)l,z 
T* (C, z) denotes the essential subtree of the complete elnbedded resolution tree 
of (C, z). 
 x««(c/s,z = z(C,z)= (f, 
o' ) ç Os,z, where x, y denote local coordi- 
nates of N at z and f e Os,z is a local equation of C. I«(C, z) is called the 
Tyuna ideal of ( C, z ). 
 x«(c)/s,z = I( C, z) Ç Os,z, where we refer for the somewhat lengthy defilfi- 
tion of I(C, z) to [GLS00, Section 1.3]. 
We call X(C) the equisingularity scheme of C and X(C) its singularity scheme. 
Analogously we call X(C) the equianalytical singularity scheme of C and X(C) 
its analytical singularity scheme. 

Throughout this article we will frequently treat topological and 
analytical singularities at the saine time. Whenever we do so, 
we will write X*(C) for Xe(C), respectively for Xe(C), and 
sinfilarly X(C) for X(C), respectively for X(C). 
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In [Los98], Propositions 2.19 and 2.20 and in Remarks 2.40 (see also [GLS00]) and 
2.41, it is shown that, fixing a point z  Y] and a topological (respectively, analyt- 
ical) type ,_q, the singularity schemes (respectively, analytical singularity schemes) 
having the saine topological (respectively, analytical) type are parametrised by an 
irreducible Hilbert scheme, which we are going to denote by Hilbz(,_q). This then 
leads to an irreducible family 

(1.5) 

Hilb(,_q) = H Hilbz(,_q). 

In particular, equisingular (respectively, equianalytical) singularities have singu- 
larity schemes (respectively, analytical singularity schemes) of the saine degree 
(see also [GLS98c] or [Los98], Lemma 2.8). The sa.me is of course true regard- 
ing the equisingularity scheme (respectively, the equianalytical singularity scheme). 
If C c Y] is a reduced curve such that z is a singular point of topological (respec- 
tively, analytical) type S, we may therefore define deg (X(S)) = deg (X(C),z) and 
deg (X*(S)) = deg (X*(C),z). We note that, with this notation, dimHilb(S)= 
deg (X (,_q)) - deg (X* (,_q)) - 2 for any z  Y], and t hus 

dira Hilb(,_q) = deg (X (,_q)) - deg (X* (,_q)). 

In applications it is convenient to replace the degree of an (analytical) singularity 
scheme by an upper bound in known invariants of the singularities. From [Los98], 
p. 28, p. 103, and Lemma 2.44, it follows that for a topological (respectively, ana- 
lytical) singularity type ç one has 

(1.6) 

deg (X«(,_ç)) < 3r(,_ç) and deg (X(,_ç)) < /(,_ç) + 2. 

1.4. Equisingular Families. Given a divisor D  Div(y]) and topological or an- 
alytical singularity types ,_ç,...,,_ç, we denote by V = VIo I (,_ç,...,,_ç) the locally 
closed subspace of IDIt of reduced curves in the linear system IDIt having precisely 
r singular points of t,,pes. ç, "" " 'ç" BY Vç = *lDl*Zç¢ct'l' --- ,ç) we denote the 
open (cf. the proof of Theorem 3.1) subset 
l/"a= {C  V I h'(P,,.ï'x(c)/r,(D)) =0} C_ V. 
Similarly, we use the notation V i = VID I (ç, .... ç) to denote the open subset 
of irreducible curves in the space V, and we set virr,re9 _-- 
V  ç V g, which is open in V *9 and in V. If a type ç occurs k > 1 times, we 
rather write kç than ç,..., ç. We call these fanfilies of curves equisingular families 
of crves. 
We say that V is T-smooth at C  V if the germ (ç\C) is smooth of the 
(expected) dimension dira IDI, - deg (X*(C)). 
By [Los98], Proposition 2.1 (see also [GK89], [GL96], [GLS00]) T-smoothness of 
V at C follows from the vanishing of H  (Y], OE¥.(c)/(C)), since the tangent space 
of V at C may be identified with H°(Y], ,.(c)/(C))/H°(Y], 0). 

lvre should hot be confused with {C e V[ h(Z,:]X.(C)/:(D)) = 0}, which is the part of 
V where V is smooth of the expected dimension. Crves in the latter subscheme are often called 
regular (c.f. [CC99]). See also Example 2.5. 
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1.5. Fibrations. Let D  Div(F) be a divisor, çl,--., çr distinct topological or 
analytical singnlarity types, and ki,..., kr  N \ {0}. We denote by / the irre- 
dncible parameter space 
/ =/(k,S1,..., kr,S-) = H Sym' (Hilb(,Si)), 
i=1 
and by B = B(kt,-çt,..., k,-çT) the nonempty, open, irreducible and dense snbspace 
B = {([XI,1,...,X,kl],..., [Xr,1,--.,X,,kr])   I snpp(Xid)rhsnpp(Xs't) = 0 
V1 <_i,s_<r, 1 <_j<_ki,.1 <t <ks}. 
Note that dim(B) does not depend on Yl; more precisely, with the notation of 
Snbsection 1.3 we have 
dim(B) = Zki. (deg (X(,_çi))- deg (X*(8/))). 
Let ns set n = Y-i=I ki deg (X(i)). We then define an injective morphism 
b = b(k,-çt,..., k,_ç)  B(kl,.ç, .... k,_ç) - Hhbr., 

([Xl,1,---,Xl,k.l,.--,[Xr,1,--.,Xr, k,.])  Ui=l Uî-=l xi,j, 

where Hilb denotes the smooth connected Hilbert scheme of zero-dimensional 
schemes of degree n on E (cf. [Los98], Section 1.3.1). 
We denote by q = çD(k$,..., k,8) the fibration of ViDl(k,.ç,..., k$) in- 
dnced by B(kl,-çl,..., k$); in other words, the morphism q is given by 

> B(kvS,..., 

c, » 
where Sing(C) = {zi,j I i = 1,...,r,j = 1,...,ki}, Xi,j = X(C, zid) and (C, zi,j) - 
Si for all i = 1,...,r,j = 1,...,ki. 
With the notation of Snbsection 1.4 note that for C  I" the fibre q-t (q(C)) is 
the open dense snbset of the linear system Iffx(c)/r(D)I consisting of the cnrves C' 
with X (C') = X(C). In particnlar, the fibres of q restricted to V  are irreducible, 
and since for C  V  the cohomology gronp H 1 (}2, ffx(c)/r,(D)) vanishes, they 
are eqnidimensional of dimension 
h°(E, ffx(c)/r(D)) - 1 = hO(}2, Or(D)) - Z ki deg (X (Si)) - 1. 
i=l 

2. THE MAIN RESULTS 

In this section we give snflïcient conditions for the irredncibility of equisingnlar 
families of cnrves on certain snrfaces with Picard nnmber one, including the projec- 
tive plane, general snrfaces in pëa and general K3-snrfaces, on prodncts of cnrves, 
and on a subclass of geometrically rnled snrfaces. 
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2.1. Surfaces with Picard Number One. 

Theorem 2.1. Let E be a surface such that 
(i) NS(E) = L-Z with L ample, and 
(ii) ]t 1 (E, C) = O, wbenever C is effective. 
Let D E Div(E), let $1,..., $ be pairwise distinct topological or analytical sin- 

(2.3) 

(2.4) 

gularity types and let kl,..., k. E N \ {0}. 
Suppose that 
(2.1) D - lç is big and nef, 
(2.2) D + K is nef, 
-k,,leg(X($i)) < .(D- tç) 2 for someO<  <_ ,l and 
i=l 
- k, deg (X($i)) 2 < "),  (D - 
Iç) 2, 
where 
,'ï = 4.x(O)+max{O,2.K.L}+6.L 2. 

Then either  " 
VID I (k,_ç,..., k$,.) is ernpty or it is irreducible of the expected 
dimension. [] 

Remark 2.2. If we set 
36c 4. \(Oz) + max{0,2- Kr.L} + 6. L 2 
7- (3c+4) 2 with c= L2 , 
then a simple calculation shows that (2.3) becomes redundant. For this we have to 
take into account that deg (X(S)) > 3 for any singularity type S. The claire then 
1 1 
follows with/3 =   -y _ < . [] 
We now apply the result in several special cases. 
Corollary 2.3. Let d > 3, let L C Fff be a line, and let $,...,$ be topological 
or analytical singularity types. 
Suppose tbat 
de(X(S,))2< -(e+3) 2. 
Then either VldLlZirr t°lrC .... , $?-) is empty or it is irreducible and T-smooth. [] 
Many authors were concerned with the question in the case of plane curves with 
nodes and cusps or with nodes and one more complicated singularity or simply 
with ordinary multiple points; cf. e.g. [Sev1], [AC83], [Har85], [Kan893], [Kan89b], 
[Ran891, [Shu91b], [Shu91a], [Bar931, [Shu941, [Shu96b], [Shu96al, [çVa196], [Los981, 
[GLS98a], [GLS98b], [Bru99], [GLS00]. Using techniques particularly designed for 
these cases t.hey get of course better results than we may expect to. 
The best general results in this case can be round in [GLS00] (see also [Los98], 
Corollary 6.1). Given a plane curve of degree d, omitting nodes and cusps, they get 
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as the nmin irreducibility coudition, where "r*($i) = "r($i) in the analytical case 
(respectively, "r* ($i.) = res(Si) in the topological case) By Subsection 1.2 we know 
that p($,) _< 2- res(Si). Thus, in view of (12), (1.3), (1.4) and of Theorem 2.1 we 
get the suflïcient condition 
r 
 («(s)+ )2 < . (d + a) 2, 
i=l 
which has the saine asymptotics. However, the coefficicnts differ by a factor of 
about 26. 
A smooth complete intersection surface with Picard lmlnber one satisfies the 
assumptions of Theorem 2.1. Thus by the Noether Theorem of the result applies 
iii pmticular to general surfaces in l: u. 
Corollary 2.4. Let Y, C  be a smooth hypersurface of degree  > 4, let H C Y, be 
a hyperplane section, and suppose that the Picard number ofE is one. Let d >  -4 
and let $1,.--, $ be topological or analytical singularity types. 
Suppose that 
X-'  (X(çi)) 2 6-(n 3-3 2+Sn-6)-,t 2 (d+4-n) 2 
/__, dg <  
-i= (n 3 - an 2 + 10, - 6) 2 " 
Then either i 
l/[dH[(81 ..... 8r) is empty or it is irreducible of the expected dzmen- 
sion [] 
We would like to thank the referee for pointing out the following example of 
redniUe fai,|is çT, (1) of noda cnvs ou ,lrfacs , e=. 
Example 2.5. If E C I?2 a is a general surface of degree n _> 4, then there is a 
finite number N > 1 of 3-tangent planes to E. However, every 3-tangent plane cuts 
out an irreducible 3-nodal curve on E, and since the Picard group is generated by 
"i(3A) 
a hyperplane section H, every 3-nodal «rve is of this form. Therefore, I IH I 
consists of N distinct points. It is thus reducible, but smooth of the expected 
dimension 
dira (VI[(3AI)) = dim IH,,- 3 = 0. 
i,.- Sing(C) have 
Note that in this situation for C  V]H ] (3AI) and z  we OE:(c)/,z = 
m 2 and thus 
h 1 (Z,,,\-(C)/E(H)) = 6 > O. 
lTirr,reglgA  
Therefore, iHI tOl = 0. The parameter space B is just Sym3(E). 
A general K3-surface has Picard nulnber one and in this situation, by the Kodaira 
Vanishing Theorem, E also satisfies the assmnption (ii) in Theorem 2.1. 
Corollary 2.6. Let  be a smooth K3-surface with NS(E) = L - Z with L ample 
and set n = L 2. Let d > O, D a dL and let $,..., $ be topological or analytical 

singularity types. 
Suppose that 

r 
Zdeg (X(,.çi))2 < 54n 2 + r2n 
=1 (lin + 12) 2 

 d 2 -n. 

Then either V1[($1,..., $) is empty or it is irreducible of the ezpected &men- 
sion [] 
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2.2. Products of Curves. If E = C1 × C2 fs the product of two smooth projective 
curves, then for a general choice of C1 and C2 the Néron-Severi group will be 
generated bv two fibres of the canonical projections, by abuse of notation also 
denoted by C1 and C2. If both curves are elliptic, then "general" just means that 
the two curves are non-isogenous. 
Theorem 2.7. Let C and C2 be two smooth projective curves of genera g and g2, 
respectively, with g >_ g2 >_ O, such that for E = C × C2 the Néron-Severi group 
fs NS(E) = C1Z @ C2Z. 
Let D E Div(E) such that D ,,« aC + bC2 with a > max{2g2 - 2, 2 - 292} and 
b > max{2g - 2, 2 - 2g }, let ,.ç,..., ,-çr be pairwise distinct topological or analytical 
singularity types and k,..., kr E N \ {0}. 
Suppose that 
(2.5) -kideg(X($i)) 2 < .(D- 
Kr.) 2, 
a--252+2 
where  may be taken from the following table with  --- b-2g+2 > 0. 

gl g2 
0 0 
1 0 
>_2 0 
1 1 
>2 >1 

1 

max{32,2} 
1 

max{24+16g1,4g1 ]. 
1 
max {32,2, } 
1 

max{24+16g1+16g2,4gla,-} 

Then either  " 
VD } (k,.ç,..., k,.ç) fs empty or it fs irreducible of the expected 
dimension. [] 

Only in the case E 
×  we get a constant  which does not depend on 
the chosen divisor D, while in the renmining cases the ratio of a and b fs involved 
in 7- This means that an asymptotical behaviour can only be examined if the ratio 
remains unchanged. 

2.3. Geometrically Ruled Surfaces. Let 7r : E = F(t) - C be a geometrically 
ruled surface with normalised bundle g" (in the sense of [Har77], V.2.8.1). The 
Néron-Severi group of E fs NS(E) = C0Z @ FZ with intersection matrix ( - ), 
where F - F fs a fibre of 7r, C0 a section of 7r with (,0t.(C0)  (,%()(1), g = g(C) 
the genus of C, e = A2E and e = - deg(e) _> -g. For the canonical divisor we have 
Kr. ~« -2C0 + (2g - 2 - e)-F. 

Theorem 2.8. Let   E -« C be a geometmcally ruled surface with e <_ O. Let 
D = aCo+bF  Div(E) witha >_ 2, b > 2g-2+-, and ifg =0, thenb >_ 2. 
Let ,.ç, .... ,.ç be pairwise distinct topological or analytical singularity types and 
 ..... r e \ {0}. 



IRREDU CIBILITY 3493 

Suppose that 
(2.6) Z kideg(X(8)) 2 < .),. (D- K) , 

where " may be taken from the following table with  - 

9 e 
0 0 
1 0 
1 -1 
>_2 0 
>_ 2 < 0  
max 

«+2 > 0. 
b÷ 2-- 2g-- -- 

1 

max{24,2} 
1 

{ rein {30÷- ÷4,40÷9 } ,?  } 
1 
max{24+ 16g,4gc} 
1 

nlin { 24÷16g--9eo,18÷16g--9eo--  } ,4go--9eo 1 

Then either irr . 
çlDI (/çl,.--,kT'-çr) is empty or if is irreducible of the expected 
dimension. [] 
Once more, only in the case g = 0, i.e., when ., - ]?C 1 × ]?¢1, we are in the 
lucky situation that the constant - does hOt at ail depend on the chosen divisor D, 
whereas in the case g >_ 1 the ratio of a and b is involvcd in % This means that an 
asymptotical behaviour can only be examined if the ratio remains unchanged. 
If  is a product C × ]1¢  , the constant " here is the saine as in Section 2.2. 
In [Ran89] and in [GLS98a] the case of nodal curves on the Hirzebruch surface 
F1 is treated, since this is just ]1¢  blown up at one point. F is an example of a 
geometrically ruled surface with invariant e = 1 > 0, a case which we so far cannot 
treat with our methods, due to the section with self-intersection -1. However, 
it seems to be possible to extend the methods of [GLS98a] to the situation of 
arbitrary ruled surfaces with positive invariant e, at least if we restrict ourselves to 
singularities that are hOt too bad. 

2.4. The Proofs. Our approach to the problem proceeds along the lines of an 
unpublished result of Greuel, Lossen and Shustin (cf. [GLS98b]), which is based on 
ideas of Chiantini and Ciliberto (cf. [CC99]). The basic ideas are in some respect 
similar to the approach used in [GLS00], replacing the "Castelnuovo-function" ar- 
guments by "Bogomolov instability'. 
We first show that the open subscheme ,rirr,reg __ loeirr,reglh ¢ 
V rr = ;ir(k8 .. k8,), and hence its closure Vr'reg in I "r, is always ir- 
reducible (cf. Theorem 3.1), and then we look for criteria which ensure that the 
complement of V r,reg in V rr is empty (cf. Section 4). For the latter, we consider 
the restriction of the morphism   V  B (cf. Subsection 1.5) to an irreducible 
component V* of V irr not contained in V ir,reg. om the fact that the dimen- 
sion of V* is at let the expected dimension dira (Vrr'reg), we deduce that the 
codimension of B* = (V*) in B is at most hl(,oEç(c)/(D)), where C e V* 
(cf. Lemma 4.7). It thus suffices to find conditions that contradict this inequality, 
that is, we have to get our hands 0n codimB(B*). However, on the surfaces that 
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we consider, the lOn-vanishing of h  (F, OE,(Ic}/r.(D)) nleallS in solne sense that the 
zero-dimellsional schelne X(Ç) is in a special position. We may thus hope to remise 
large parts X of X(C) on curves  of "Sllmll degree" (i = 1,..., m), which would 
impose ai least #X - dira Ilt conditions on X(Ç), giving rise to a lower bound 
' #Xï-dim I1 for codims(B*). The Xï's and the ,'s are round in Lemma 
4.1 with the aid of certain Bogolnolov unstable rank-two bundles. It thus finMly 
relnains (cf. Lemlnata 4.3, 4.4 and 4.6) to give conditions that imply 
 #Xï -diln [i] > h(.x(c)/(D)). 
These considerations lead to the following proofs. 
Proof of Theorem 2. 1. X nlay a.ssulne that V i is nonempty. By Theorem 3.1 it 
suffices to show that V i = 'irr,reg. 
Suppose the contrary, i.e., there is an irreducible curve Co  V i  Vi««o; in 
particular, h.(.oEx-,,/(D)) > 0 for Xo = X(Co). Since 
,teg(.ko) 
i=1 
and EzN ( deg( X0,z ) ) 2 = EiL1 k i deg(X ( Si ) ) 2, thP aSSulnpt ions (0)-(3) of Lemma 
4.1 and (4) of Lelnlna 4.3 are fulfilled. Thus Lemlna 4.a inlplies that C0 satisfies 
condition (4.19) in Lelmna 4.7 which it cannot satisg" bv the smne lemma. Thus 
we bave derived a contradiction. 
Proof of Theorem 2.7. The Sulnptions on a and b ensure that D - h- is big and 
nef and that D + Ks is nef. Thus, once we know that (2.0) implies condition (3) in 
Lemma 4.1 we tan do the saine proof  in Theoreln 2.1. just replacing Lemma 4.3 
by Lemma 4.4. 
For condition (3) we note that 
 :i deg (X(i))_ < ki" (deg (X(Si))) 2 
i=I i=1 

Proof of Theorem 2.8. The proof is identical to that of Theorem 2.7, just replacing 
Lenlma 4.4 by Lenllna 4.6. [] 

2.5. Sollle Remarks. What are the obstructions to our approach? 
First, the Bogolnolov instability does hot give lnuch inforlnation about the curves 
Ai apart from their existence and the fact that they are in SOlne sense "slnall" com- 
pared with the divisor D. We are thus bound to the study of surfaces where we 
have a good knowledge of the dimension of arbitrary complete linear systelns. Sec- 
ond, in order to derive the above inequality, many nasty calculations are necessarv 
which strongly depend on the particular structure of the Néron-Severi group of the 
surface, that is, we are restricted to surfaces where the Néron-Severi group is hot 
too large and the intersection pairing is not too hard (cf. Lemmata 4.3. 4.4 and 
4.6). Finally, in order to ensure the Bogomolov instability of the vector bundle 
considered throughout the proof of Lemma 4.1 we heavily use the fact that the 
surface E does hot contain any curve of negative self-intersection, which excludes 
e.g. general Hirzebruch surfaces. 
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If the number of irreducible curves of negative self-intersection is not too large, 
one might overcone this last obstacle with the technique used in [GLS98a]. That 
is, we would bave to show that under certain additional conditions the singular 
points of the considered curves could be independently moved; in particular, they 
could be moved off the exceptional curves- more precisely, the subvariety of V iTr 
of curves whose singu]ar locus does hot lie ou any exceptional curve is dense in 
V i''. For this, one basically just needs criteria for the existence of "snmll" curves 
realising a zero-dimensional scheme slightly bigger than the equisingularity scheme 
(respectively, the equianalytical singularity scheme) of the members in I . For 
example, in the case of curves with r nodes, that means the existence of curves 
passing through r arbitrary points and having nmltiplicity two in one of them. 
In Section 3 we not only prove that V i''''eg is irreducible, but also that this 
indeed renmins true if we drop the requirement that the curves should be irreducible, 
i.e., we show that V « is irreducible. However, unfortunately our approach does 
not give conditions for the enptiness of the complenent of V e, and thus we cmmot 
say anything about the irreducibility of the variety of possibly reducible curves in 
IDIt with prescribed singularities. Thc reason for this is that iu the proof of Lenmm 
4.1 we use the Bézout Theorem to estimate D - Ai. 

3. V irr'reg |S IRREDUCIBLE 

We now show that V i«eg is always irreducible. We do this by showing that 
under  : V -- B every irreducible component of V i''«eg is smooth and maps 
dominant to the irreducible variety B with irreducible fibres. 

Theorem 3.1. Let D  Div(E), ç,..., Ær be pairwise distinct topological or ana- 
lytical singularity types and k , . . . , k  N \ {0}. 
lfirr,reg(I  . , 
If ID[ ç,_q,.. k,.ç,.) is nonempty, then it is a T-smooth, irreducible, open 
subset of i .  
D] ( lsl' " " " ' rSr) of dimension dira ID,, - 2,=, k deg (X*(8i)). 

eirr,reg[L  ., reg 
Proof. Since Vin I ,lOa,-. kS) is an open subset of Vin I (]çlSa,..., krSr) -- 
V «e, it sufiïces to show the claire for V «e. 
Let us consider the following maps from Subsection 1.5: 

and 
l, = ¢( k,Sa, . . . , k.,-% )  B( k,S , . . . . k,.,S,. ) ------> Hilb.. 
Step 1: Every irreducible component V* of V reg is T-smooth of dimension 
dira IDI- -= k deg (X*(,Si)). By [Los98], Proposition 2.1 (c2), V* is T-smooth 
at any C e V* of dimension dira IDI - deg (X*(C)). since h a (, OEx*/r.(D)) = 0. 
Note that deg (X*(C)) = -'-a kideg(X*(,Si)) only depends on k,S,...,k.,S. 
(cf. Subsection 1.3). 
Step 2: V eg is open in V. Let C Ç Veg; then h(2, OE:(c)/E(D)) = 0. Thus 
by semicontinuity there exists an open, dense neighbourhood U of X(C) in Hilb 
such that hl(E,,.TWr(D))= 0 for all Y Ç U. But then tP-a (¢-a(U)) Ç V «e is an 
open neighbourhood of C in V, and hence V Teg is open in V. 
Step 3:  restricted to any irreducibte component V* of V eg is dominant. Let 
V* be an irreducible component of V «e and let C  V*. Since -t((C)) is an 
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open, dense subset of l7,«c)/(D)[t and since h 1 (E,,.Tx(c)/E(D)) = 0, we have 
dim - ((C)) = h°(E,,.Tx(c)/E(D)) - 1 = dim IDI, - aeg (X(C)). 
By Step 1 we know the dimension of V* and by Subsection 1.5 we also know the 
dimension of B. Thus we conclude 
dim(V*) = dimV*-dim-((C)) 
= (dira IDI - deg X*(C)) - (dira IDI - deg X(C)) 
= deg(X(C))-deg(X*(C)) = dimB. 
Since B is irreducible, (V*) nmst be dense in B. 
Step 4: V  is irreducible. Let V* and V** be two irreducible components of V 
Then (V*)  (V**) ¢ 0, and thus some fibre F of  intersects both, V* and 
V**. However, the fibre is irreducible and by Step 1 both V* and V** are smooth. 
Thus F must be completely contained in V* and V**, which implies that V* = V**, 
since both are smooth of the saine dimension. Thus V 

4. THE TECHNICAL DETAILS 
The following lemma is the heart of the proof. Given a curve C E IDI/, whose 
(analytical) singularity scheme Xo = X(C is special with respect fo D in the sense 
that h  (E, ffXo/(D)) > 0, provides a "small" curve & through a subscheme Xï 
of Xo, so that we can reduce the problem by replacing Xo and D by Xo " & and 
D - &l respectively. We can of course proceed inductively  long  the new 
zero-dimensional scheme is again special with respect to the new divisor. 
In order fo find & we choose a subscheme Xï Ç Xo that is minimal anaong 
those subschemes special with respect fo D. By Grothendieck-Serre duality, 
H(E,xï/(D))  Ext  (xï/(D- K),O) 
and a nontrivial element of the latter group gives fise to an extension 
0  Oz  E  Jxï/(D - Kz)  O. 
We then show that the rk-two bundle E is Bogomolov unstable and deduce the 
existence of a divisor Aï such that 
(, ï, (-  - ï))  0, 
that is, we find a curve A e IJx/z(D - Kz - Aï)l ,. 
Lemma 4.1. Let E be a surface such that any cue C C E is nef. (Assumptzon 
(.)) 
Let D  Div(E) and Xo C E a zero-dimensional scheme satisfying 
(0) D - Kz is big and nef, and D + Kz is nef, 
() 3 Co  IDI, irreducible" Xo C Co, 
() h(Z,Y./(D)) > o, ..d 
1 
(3) deg(Xo) < ç-(D - Kz)  for some 0 < ç  . 
Then there ezist cues A1 .... , A C E and zero-dimensional locally complete 
intersections Xg ç Xi-  Ai for i = 1,...,m, where Xi = Xi-1 " Ai for i = 
1,..., m, such that 
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and for i = 1,...,m 
(b) h'(E"]xï/2(D - Y'}k:,-' Ae)) = 1, 
 2 
(c) D.A i >_ deg(Xi_i nai) 2 deg (Xï)  (D- h's- Zk=, ak).ai  a  O. 
i >0, 
(d) ( - u - Z=  - ) 
 Ai).H 0 for ail H Div(Z) ample, and 
(e) (D -  - =1 A - > e 
(f) D - Ks - k=l Ak is big and nef. 

Moreover, it follows that 

(4.1) 

 ( ) 
0<_ ¼(D-Kr,)2-Zdeg(Xï) <_  
i=i =1 

Pro@ We are going to find the schemes Ai and X/° recursively. Let us therefore 
suppose that we have already round A,,..., Ai_ , and Xï,... ,Xî_, satisb'ing (b)- 
(f), and suppose that still h(Y]',J]x,- 1/Y]( D -- Ek=''--' Ak) ) ) O. 
We choose minimal Xî ç Xi-1 su«h that h 1 (E,]xIs(D i--1 
sp1: ,(s, «/(- ', )) = 1, i.., (u) fla. 
Suppose it w strictly larger than one. By (0), respectively (f), and by the Kawa- 
mWUw W.. Uom w .  (,( - )) 
-=A =0. 
Thus Xî cannot be empy, that is, deg (Xï)  1 and we may choose a subscheme 
Y C Xî of degree deg(Y) = deg (Xî) - 1. The inclusion OEp  y implies 
(,,(_ -, )) < _ -, 
=, _ (s,«.(v :, )) ,,d  ¢t 
sequences of Y and Xï thus lead to 

h'(E,j]y/s(D- ZikI ', A.)) >_ h'(E,j]'x.o,/s(D- Z'?/, A)) -1>0 

contradicting the minimality of Xî. 
Step 2: deg (Xî) <_ deg(X0) - Ea=, deg(Xa_l 
The case i = 1 follows from the fact that Xï C_ X0, and for i > 1 the inclusion 
Xî ç Xi-, = X-2  Ai-, implies 

deg (Xï) <_ deg(Xi_2  Ai-,) = deg(Xi_2) - deg(Xi_2 r 

It thus sufiïces to show that 

i--1 
deg(Xi-2) - deg(Xi_2 ç Ai-,) = deg(Xo) - Z deg(Xk_, ç Ak). 
k----1 

If i = 2, there is nothing to show. Otherwise Xi_ 2 ----" Xi-3 " Ai-2 implies 

deg(Xi_2) - deg(Xi_2 CI Ai-l) 
= deg(Xi_3  Ai-2) - deg(Xi_2 CI 
= deg(Xi_3) - deg(Xi_3 CI Ai-2) - deg(Xi-2 rq Ai-l) 

and we are done by induction. 
Step 3: There exists a "suitable" locally free rank-two vector bundle Ei. 
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By the Grothendieck-Serre duality we have 
i--1 
0 ¢ HI(E,,_Txï/z(D - Ek=l Ak)) ' Ext 1 (x-ï/z(D - h'z- 
That is, there exists an extension 
The mininmlity of Xï implies that Ei is locally Dee and hence that Xï is a locally 
complete intersection (cf. [Laz971), 
(4.3) Cl(Ei)=D--h'E--k aild c2(Ei)=deg(Xï). 
k=l 
Step 4: Ei is Bogomolov unstable. 
According to the Bogomolov Theorem we only bave to show that c 
(cf. [Bog79] or [Laz97], Theorem 4.2). Since (4- 1)-(D - K)   0 by (a) and 
since & OE 0 by Assumption (,), we de&me: 
i--1 
4c(Ei)= 4deg(X) < 4deg(X0)-4deg(Xk_&) 
--Step 2 
k=l 
<{a)/{) 4(D-K) -2Ak. D-K- 
k=l j=l 
( i-1 )2 -1 
= D- K - A + (4- 1)-(D- K) 
k=I k=I 
 D-'z- = c(&) . 
k=l 
Step g: Find &i. 
Since Ei is Bogomolov unstable, there is a 0-dimensional schenm Z ç N and a 
&  Div(N) such that 
(4.4) 0  O(&ï)  Ei  z/ (D - K -   - )  0 
is exact and such that 
(d') (2Aï- D+ h'z+ E=I A) =  C1(«i)2--4.C2(«i) > 0, and 
i--1 
(e') (2&ï - D + K + =1 &k)-H > 0 for all H  Div() ample. 
Tensoring (4.4) with O( - &ï) leads to the following ext sequence: 
  -  - 2ï  0, 
(4.5) 0 O= Ei( Aï)  J&/z D- K - = 
n w «« t o(,«( _ )  . 
Now tensoring (4.2) with O(- &ï) leads to 
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By (e'), and (0), respectively (f), 
i--1 
-- < -(»-  - }2 ). -< 0 
k=i 
for an atnple divisor H. Hence -Aï cannot be effective, that is, H°(E,-Aï) -- O. 
But the long exact cohomology sequence of (4.6) then implies 
In particular, we ,nay choose Ai  [OE/(D- '- :, A- 
Step 6: A, satisfies (d)-(f). 
We note that by the choice of Ai we have the following equivalences: 
i 
(4.7) ï , D- L- -  a, 
k=l 
i--1 i 
k=l k=l 
Thus (d) and (e) is a reformulation of (d') and (e'). 
Moreover, since (A - A,).H > 0 for any ample H, we have (A? - Ai).H  0 
for any H in the closure of the ample cone; in particular, 
(4.9) Aï.H  Ai.H  0 for all H nef, 
since A is effective. Finally, since bv sumption (*) any effective divisor is nef, 
we deduce that Aï.C 2 0 for any curve C, that is, Aï is nef. In view of (4.7) for 
(f) it remains to show that (Aï)2 > 0. Once more taking into account that Li is 
nef by (*) we have by (d'), (4.8), and (4.9), 
(a)  = (a - a,)  + (a? - a).a, + a.a, > 0. 
Step 7: & satisfies (c). 
We would like to apply the Bézout Theorem to Co and &. Thus suppose that the 
irreducible curve Co is a component of & and let H be any ample divisor. 
Applying (d) and the fact that D + tç is nef by (0), we derive the contradiction 
0 (Ai-Co).H<-. D + h' +  Ak .H  - . (D + K).H  O. 
Since Xi-1 ç Xo c Co, the Bézout Theorem therefore implies 
D.Ai = Co.Ai  deg(X.i_t  Ai). 
By definition X ç Xi_ and Xî c Ai. Thus 
deg(Xi_  &)  deg (Xï). 
By assumption (*) the curve Ai is nef and thus (4.9) gives 
D-Kz- Ak .Ai=A i.A2A, 0. 
k=l 
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Finally from (d') and by (4.3) it follows that 

(o_ ,.) > «(<)- 4. «(<)= (a? + )- 4. deg (x°), 

and thus deg (Xï) _> Aï.Ai. 
Step 8: A[ter a finite number m of steps, h'(E,,ffx,,/E(D- -'im__l Ai)) 
As we have mentioned in Step 1, deg (Xï) > 0. This ensures that 

=0. 

deg(Xi) = deg(Xi-1) - deg(Xi_l ç Ai) _< deg(Xi_) - deg (Xï) < deg(Xi_), 

i.e., the degree of Xi strictly decreases each time. Thus the procedure must stop 
after a finite number m of steps 
Step 9: It remains to show (4.1). 
By a8sumption (*) the curves Ai are nef; in particular, Ai.Ai >_ 0 for ail i,j. Thus 
(c) implies 

m) 
+ 

But then, taking condition (3) into account, 

[] 

It is our overall aim to compare the dimension of a cohomology group of the form 
Hl(E,.ffXo/:(D) with some invariants of the Xï and Ai. The following lemma 
will be vital for the necessary estimates. 

Lemma 4.2. Let D E Div(E) and let Xo C E be a zero-dimensional scheme such 
that there exzst curves AI,..., A m C Y], and zero-dimensional schemes Xï C_ 
for i = 1,...,m, where Xi = Xi- : Ai for i = 1,...,m, such that (a)-(f) in 
Lemma 4.1 are fulfilled. 
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Then: 

m 
i=l 
m 
i=1 
m 
i=1 

(1 + deg(Xi-1 hA/)- deg (Xï)) 

i 
(Ai'(Kr.+ZAk)+I)- 
k----1 

Proof. Throughout the proof we use the following notation: 
çi JXi_lrA,/A, D - k:i A k and çï : «Xï/A, (D - Zik211- Ak) 
for i = 1,...,m, and for i = 0 .... ,m, 
Since X+ = X : Ai+, we have the following short exact sequence: 
"*+1 
(4.10) 0  i+  i  Çi+l  0 
for i = 0,..., m - 1 and the corresponding long exact cohomology sequence 
(4.11) 
0  H°(, +)  H°(,)  H°(,ç+)  H(,+I) 
0 = H(, ç+l)  H(, )  H(,+I)  H(, ç+l)  H(,) 
m 
Step 1: h(,i) j=i+h(,çj) fori=0,...,m-1. 
We prove the claim by descending induction on i. om (4.11) we deduce 
0 = H(, ) H1(,_1) H(,ç), 
which i mplies h  (, _ 1)  h  (, ç) and thus proves t he ce i = m- 1. 
We may therefore sume that 1 < i < m - 2. Once more from (4.11) we deduce 
a = h°(,+) - h°(,) + h°(,ç+)  0 

and 

b = h2(E,i+) - h2(E,i) >_ O, 

hl(E, Çi+I) ÷ hl(,gri+l) - a - b _< h(E,ç+) + hl(E, gri+) 
m m 
< hl('Çi+l)÷ Z hl('ÇJ) = Z hl('ÇJ)" 
--Ind. 
j=i÷2 j=i÷ l 
Step 2: hl(Ai, Çi) =- hO( Ai, Çi) -- X(OA, (D - _11Ak) ) ÷ deg(Xi_ 1 fil Ai). 
We consider the exact sequence 

0 ---> çi ---> OA, D -- k=lAk ---> OX,_IC)A,/A, D - ----> O. 

The result then follows from the long exact cohomology sequence. 



3502 THOMAS KEILEN 

Step 3: h°(Ai,Çï)- X ((.9/x, (D - -._11Ak)) = hl(Ai,Çï)- deg(Xï). 
This follows analogously, replacing Xi- by X°i, since Xï = Xï r-i Ai. 
- Ek= a)) = 1. 
Step 4: h(a,çï) <_ h  (E, ,7x,0/r. (D 
Note that Xï  a, = 0, and hence ,.Txo,:a,/r. = Or.. We thus have the following 
short, exact, sequence: 

(:) ( -) 
(4.12) 0Or. D- =A» oE.qo/r. D-Z=I/k Çï0. 

i 
By assulnption (f) the divisor D - Kr. - E=I Ak is big and nef and hence 

i 
k=l 

Tlms the long exact cohomology seqnence of (4.12) gives 

and 
However, by assumption (b) t.he latter is just one. 
Step 5: hl(A_.,Çi) _< 1 + deg(X,_ r-i A)- deg (Xï). 
We note that Çi ¢- Çï, and thus h°(A,ç) <_ h°(A.Çï). But then 

h'(/ki,Çi ) < h(/ki,Ç/°) - deg (X, °) + deg(Xi_, I-lai) 
--Step 2/3 
< 1-deg(Xi °) +deg(Xi-, OAi). 
--Step 4 
Step 6: Finish the proof. 
Taking into account that h  (E, Ç) = h 1 (A,, Ç), since Çi is concentrated on Ai, the 
first inequality follows from Step 1. while the second inequality is a consequence of 
Step 5, and the last inequality follows froln assumption (c). [] 

In Lemmata 4.3. 4.4 and 4.6 we consider special classes of surfaces which allow 
us to do the necessary estimates in order to finally derive 
m 
Z (#Xï - dim[Ai[) > h(E,,JXo/r,(D)). 

We first consider surfaces with Picard number one. 

Lemma 4.3. Let E be a surface such that 
(i) NS(E) = L.  and L is ample, and 
(ii) h  (E, C) = 0. whenever C is effective. 
Let D  Div(E) and let Xo C E be a zero-dimensional scheme satisfying (0)-(3) 
from Lemma 4.1 and 
(1 +'Z-) 2.L 2 
(4) E..s (deg(Xo,..))  < ". (D - -s), .ere - = -(o)+o,«.+.- 
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Then, using the notation of Lemma 4.1 and setting Xs = Ui= X° 
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ha(2, oE,¢o/r,(D)) +  (h°(2, Or,(A))- 1) < #Xs. 

Proof. We fix the following notation: 

D,«d.L, K,«n-L, Ai,-«6i-L, andl= x/>0. 

(1 ÷ Vï:-) 2 
Furthermore, we have -y = 4 , where 

4- )/(Or) + max{0,2- K.L} + 6. L 2 

4" L 2 

{ x(Or.) 
: --+ , 
----- + , 

Step 1: By (i) E satisfies assumption (*) of Lemlna 4.1. 
Step 2: Ei=I 5i" l _< (a-,).12 (d-).t deg(Xs), bv (4.1). 
m 1(( m m ) 
Step3: h(r,y0(»)) < ('E--15i) +3 E--15) + E,--15 " +'- 
By Lemnm 4.2 we know that 

+ ]   

12 +m. 

m( ) l m 2 .l m 
Step4: E=I h°(E,O(A)) -1 -<m'0(Om)-l)+Y'E=6  "E,=15  
Since A is effective by (ii), hl(E, Ai) = 0. Hence by Riemann-Roch 

=,.((o)-i)+.6-  .. 
z=l i----1 

Step 5: Finish the proof. 
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In the following consideration we use that deg(Xs) < deg(X0) </3. (d - )2./2: 

hl(-'ffxo(D)) ÷ E ( h0(}-' ('gE(/i)) -- 1) 

m 12 
< m.x(Or)+l 2  2 n" 
--Step 3 / 4 " {i -I- 2 

2 
i----1 i=1 
2 
4 - deg(Xs) 

( 2  deg(Xs)) 
(d - n) . l + xT(-d7-- :  - 4  deg(Xs) 

The second class of surfaces that we consider are products of curves. 
Lemma 4.4. Let C1 and C2 be two smooth projective curves of genera gl and 
respectively, with 91 OE 92 >-- O, such that for E = C1 x C2, the Néron-Severi group 
is NS(E) = CZ @ C2, and let D  Div(E) be such that D ,,« aC + bC2 wzth 
a > max{2g2 - 2, 2 - 292} and b > max{2g - 2, 2 - 29}. Suppose moreover that 
Xo C  is a zero-dimensional scheme satisfying (1)-(3) from Lemma . 1 and 
(4) Eez (deg(Xo,z)) 2 < 7" (D - Kz) 2, 
where 7 may be taken from the table in Theorem 2. 7. 
Then, usin9 the notation of Lemma 4.1 and setting Xs = Uim=l Xï, 
m 
hl(E,xo(D)) 
E 
+ (h°(E, OE(Ai))- 1) < 
#Xs. 
Proof. Then Kz a (292 - 2)  C + (291 - 2)  C2 and we fix the notation: 

Ai « aiC + bC2, 
Step 1: E satisfies assumption 
on a and b we know that D - 
4.1 is fulfilled as well. 
Step 2a: (x). Eirn=l bi + (- 

nl=a-2g2+2 and 2=b-29+2. 
(*) of Lemma 4.1. Moreover, due to the assumptions 
KE is nmple and D + K is nef, i.e., (0) in Lemma 

)" i= ai <_ deg(Xs). 
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(4.13) 
and 

Let us first notice that the strict inequality "<" in Lemma 4.1 (e) for ample divisors 
H cornes down to "<_" for nef divisors H. We may apply this for H = C1 and 
H = C2 and deduce the following inequalities: 
0_< D-/fr- A-A i .C1=2- b-b 
k=l k=l 

(  )  
(4.14) 0<_ D-Kz- Ak--Ai .Cee=- ak-ai. 
k=l k=l 
For the following consideration we choose i0, j0 E { 1,..., m} such that aio >_ ai for 

ail i = 1, .... m and bjo >_ bi for ail j = 1 .... , m. Then 
(4 15) g >_ 2ai and  >_ 2bi 
for all i,j = 1,... ,m; finally (4.13)-(4.15) lead to 

deg(Xs) = Zdeg (Xï) > 
--Lemma 4.1 (c) 
i=1 

m(±) 
Z D- hz - 
i=1 k=l 

i=1 

Step 2c: -,=l ai < (D-K,) " (deg(Xs)) 2, if -i=l bi = 0, 
-- / 8 
(D-K)" (deg(Xs)) 2, otherwise. 

If -i= bi = 0, then the same consideration as in Step 2a shows that 
deg(Xs) _> 2- a, > 0, 
i=1 

, " Z ai <_  " ai _< (deg(X8)) 2. 
i=1 i=1 

and thus 

.Ai 
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If }-= bi -7/: O, then »ve are done by Step 2b. 
{  
Step 2d: m= bi < °°(D-Içs)2" (deg(XS))2' if = ai = O, 
-- 8 , otherwise. 
(D-K)  " (deg(X8)) 2 
This is proved in the smne way as Step 2c. 
Step 3: h (Y;, 27xo(D)) _< Ei=l ai -i=1 bi+(2gl-2) Ei=l ai+(2g2-2) Yi= biq- 
The following sequence of inequalities is due to Lemnm 4 2 and the fact that 
&.&j >_ 0 for any i,j  {1,...,m}: 

Step 4: We find the estimate , (h°(E,O(&,)) - 1) _< ff, where 

m m m 
Yi= ai  Yi= bi + Y= bi, if gl = 1, g2 = 0, 
-i=1 ai  -i= bi - m, if gl ---- 1, g2 = 1,  i 0  aiobio > O, 
= 
-i=1 ai -t- -i=1 bi - m, if gl = 1, 92 = 1, V i  aibi = 0, 
Yi= a . Yi= bi + Yi= ai + Yi= bi, otherwise. 

In general h ° (Y;, O(&)) _< ab + a + b + 1, whereas if g =- 1, g2 = 0, »ve have 
h ° (Y;, O(&)) = ab +b + 1. It thus only remains to consider the case g = g2 = 1, 
where ,,ve get 

i=1 ai ,b >0 a,=0 b,=0 

If always either ai or bi is zero, »ve are done. Otherwise there exists some i0  
{ 1 .... , m} such that a o ¢ 0 ¢ bio. Then looking at the right-hand side we see 

m 
i l ai ,bi >0 

Tri m 
a=O bi=O i=1 i=1 

b i . 

Step 5: Finish the proof. 
Using Step 3 and Step 4, and taking m _< YI ai + bi into account, »ve get 

m m 
3" -i=1 ai" -i=1 bi, 
m m 
a. î=,  . =, » + = , 
m m 

if g = O, g2 : O, 
if g = 1, g2 : O, 
if g > 2, g2 k O. 
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For the case gl ---- g2 = 1 we take a closer look. We find at once the following upper 
bounds f" for hl (S, «xo(D)) + Eim=l (h°(S, Or.(Ai))l 
,, = 3- OEi=I ai" OEi=l bi, if 3 io  aiobo # O, 
2. E= a- E= b + E=I a + E---I b, f V " a,b = 0. 
rrt rrt m OEi=X b ---- 0, 
Considering now the cases =1 ai # 0 # i=1 b, -:i=1 ai = 0 and 
we can replace these by 

/ m m 
4" Ei=l ai" Ei=l bi, 
m 
" <- ' = F=I ai, 
m 
OEi=l b, 
Applying now the results of Step 2 in all cases we get 
( ) 
2 
.(D_gs)  deg(Xs, 
- .(D-Ks) 
< #Xs 
_ .(D_K)  deg(Xo,) 2 < 
z (4) 

if Eim__l a i # 0 # Eim=l bi, 
m 
if }ï:i=l b = 0, 
if " 
Ei=I ai = O. 

1 . (deg(Xs)) 2 
/.(D-K) 2 

 deg(Xs,) 2 

Remark 4.5. Lemma 4.4, and hence Theorem 2.7 could easily be generalised to 
other surfaces E with irreducible curves C1, 
with intersection matrix (ï 0  ) once we have an estimate similar to 
h°(E, aC1 + bC2) <_ ab + a + b + 1 
for an effective divisor aC1 + bC» 
With a number of small modifications, we are even able to adapt it in the follow- 
ing lemma in the case of geometrically ruled surfaces with non-positive invariant e 
although the intersection pairing looks more complicated. 
The problem with arbitrary geometrically ruled surfaces is the existence of the 
section with negative self-intersection, once the invariant e > 0, since then the proof 
of Lemma 4.1 no longer works. 

In the following lemma we use the notation of Subsection 2.3. 

Lemma 4.6. Let 7r  E -- C be a geometrically ruled surface with invariant e <_ 0 
and g = g(C), and let D E Div(E) be such that D ,,« aCo + bF with a >_ 2. 
b > 2g - 2 + -, and if g = O, then b >_ 2. Suppose moreover that Xo C E is a 
zero-dimensional scheme satisfying (1)-(3) from Lemma 4.1 and 
(4) Ees (deg(X0,z)) 2 < " I ( D l   )" 2 ' 
where 3  may be taken from the table in Theorem 2.8. 
m 
Then, using the notation of Lemma 4.1 and setting Xs = Ui=l x9 
m 
hl('7X°(D) + E (h°('OE(Ai)) - 1) < X S. 
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Pro@ Remember that the Néron-Severi group of E is generated by a section C0 
of r and a fibre F with intersection pairing given by ( -  ). Then h r. "« -2C0 + 
(2g - 2 - e)  F and we fix the notation: 

Ai  « aiCo + b'iF. 

Note t.hat then 
ai _> 0 and bi := b'i - ai _> 0. 
Finally, we set. 1 = a + 2 and 2 = b+ 2 - 2g -  and get 
(4.16) (D- Kr.) 2 = -e. (a + 2) 2 + 2. (a + 2). (b + 2 + e - 2g) = 2. 1  n2. 

Replacing the equations (4.13) and (4.14) by 

(  ) ± 
(4.17) 0_< D-Kz- Ak--Az .(Co+F)=n2- bk-bi 
k=l kl 

and 

(4.18) 

0_< D-Kz- Ak-Ai .F=n- a-ai, 

the assertions of Step 1 to Step 2c in the proof of Lemnm 4.4 remain literally true. 
m 32a 2 32 
Step 2d: ai <- (D-K) (deg(X8)) and b  .(D-N) (deg(X8)) 2 
This follows from the following inequality with the aid of Step 2a and (4.16): 

Step 3: h (E,x(D)) < 2-Ei ai. Eî= bi+(2g-2)-Eî= ai-2-Eil bi+m 
is proved as Step 3 in Lemma 4.4. 
Step 4a: If e = 0, we find the estimate 

m m m m 
Eai'Ebi+Ebi-m, if 9=l,Ebi#O, 
i=l i=l i=l i=1 
Eai" Ebi+Ebi=0, if9= 1, Ebi=O, 
i=l i=l i-l i=l 
E ai" E bi + E ai + E bi, for g arbitrary. 
i=1 i=i i=i i=i 

We note that in this case bi = bi and that bi = 0 thus implies ai > O. But then 

{ aib+bi, ifg=l, bi>O, 
h°(E, Or.(Ai)) < aibi+bi+l=l, if9 = 1,bi = O, 
aibi + ai + bi + 1, otherwise. 

m 
The result.s for g arbitrary, respect.ively, g = 1 and Ei=I bi = 0 thus follow right 
away. If, however, some bi > O, then i¢j aibj >_ bio i¢ ai >_ # {bi I bi = O} 
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and hence 

m m 
h°(, O(A{)) _< -a,b{ + -b, + #{b{ I b, 
i=l i=l 
m m m m m m 
i=1 i=1 i=1 i#j i=1 i=1 

Step 4b: Ife < O, we give several upper bounds for fi = ,i= (h°(,O.(Ai))-l)  

If g is arbitrary, the claire follows since a thorough investigation leads to 

9e 2 
h°(N,Or(Ai)) <_ aibi +a, +b, + 1-  .a i 

and 
  aibi + ai + b + 1 - 9e 2 __ 1 
h°(r,o(/x)) <   .,:, . 
Ifg=l, thene=-I andb=b'+ « 
3, and we aïe done since 

bi 2 . 

a,(a,+l) b(b'i-1 ) 
h°(N, Or(Ai)) <_ab'i+b'i + l +  +-  
1 2 1 1 
- 2- ! " aibi + ½ " bi  + g . a i +  . ai +  . bi + l. 

Step 5: In this last step we gather the information from the previous investigations 
and finish the proof considering a bunch of different cases. 
m 
Using Step 3 and Step 4 and taking i=1 ai + bi < m into account, we get the 
following upper bounds for ri'= bi (, ,.]'x0 (D)) A- im=l (hO(,Or.(Ai)) -- 1]: 
\ - - 

m m m 
i=l i=1 i=l 
   ( ) 
3  ai  bi A- 2 9  ai -- - ai , 
i----1 i=1 i=1 i=1 
   () 
9 9e 
i=1 i=1 i=1 
m m 
3  ai  bi, 
i=1 i=1 
m 
m<_ai, 
i=1 
  () 
5 1 bi 
,b+ 
i=1 i=1 i=l 

if e = 0, 

if e<0, 

2 
(£bi) , ife<O, 
m 
if e = 0,g = 1,  b # 0, 
i=1 
m 
if e = 0,9 = 1,  b = O, 
i=l 
( )  
1 5 
+ g ai + -  ai, if e < 0, g = 1. 
i=1 i=1 
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Applying now Steps 2b-2d we end up with fl'.(D-K:): 
(deg(Xs))  7" We thus finally get 

])1 (Y', .TXo(D)) ÷ 2 }'O(,Or.(,/i) ) -- 1 =  
i=1 
_ 1 #Xs 
-- 7-(D-gV" ( E deg(X,)) 2 - < 7-(D-gV 
zGE 
< #Xs  deg(Xo )2 < #Xs. 
-- 7"(D-K)  " (4) 
z6E 

1 . (deg(Xs))  
-,.(D-K) 2 
 y deg(Xs,z) 2 
z6E 

It relnains to show that the inequality that we derived cannot hold. 
Lemma 4.7. Let D 6 Div(E), let ,SI .... , St be pairwise distinct topological or ana- 
.... IIirr, re9[ 
lytical singularity types and let kl, kr 6 N{0}. Suppose that 
kS ) is nonempty. 
rr .   IIirr're9[l" ¢ , krr) 
Then there exists no curve C 6 D 
such that for the zero-dimensional scheme Xo = X(C) there exist curves Al,..., 
A C E and zero-dimensional locally complete intersections Xï ç Xi- for i = 
1,.. m where X, Xi_ 1  A, for i 1,.. m, such that Xs 
 , , = = ., = Ui= Xï satisfies 
m 
(4.19) bi (r,xo(D))+  (b°(r, oE(a,))- 1) < Xs. 
Pro@ Throughout the proof we use the notation 
= Dl (kS,...,kS) and 
-- V[D ] kl°l, "'" , 
Suppose there exists a curve C  V i  V i«e satisfying the sumption of the 
lemma, and let V* be the irreducible component of V i containing C. Moreover, 
let Co  V *'e. 
In the following we consider the morphism from Subsection 1.5, 
 : ]D(klSl,..., krSr) " D(kS,..., 
Step 1: h°(Z, oEx-(Co)/(D)) = h°(Z, oEx-(c)/(D)) - hl(Z,x(c)/(D)). 
By the choice of C0 we have 
0= Hl(,Y-.¢Co/())  HI(,O())  Hl(,Ox.(Co()) =0, 
and thus D is non-special, i.e., hl(Z,O(D)) = 0. But then 
°(, &-(«o/(D) ) = h°(,y(c/(D)) - 
Step 2: hl(Z,x(c)(D))OE codims ((V*)). 
Suppose the contrary, that is, dim ((V*)) 
dim( B) - hl (Z, oEx-(c)/( D) ). 
Then 
by Step 1 and Theorem 3.1, 
dim(V*) N diln((V*))+dim(-((C))) 
< dim(B)- h(Z, oE(c)/(D)) + h°(Z, oE(c)/(D)) - 1 
= dim(B) + h°(Z,X(Co)/(D)) - 1 = dira (Vi'eg). 
However, any irreducible component of V i h at let the expected dimension 
dira (Vi««9), which gives a contradiction. 
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Step 3: codimB (tP(V*)) _> #Xs - ,i=ldi,n[A,,. The existence of the sub- 
schemes Xï C X(C)ç A, in,poses at least #Xï- dimlAilt conditions on X(C) 
and increases thus the codimension of tP(V*) by this number. 
Step 4: Collecting the results we derive the following contradiction: 

h(N,..Tx(c)(D)) > 
--Step 2 

rn 
> #Xs- 7, dimlAilt > 
--Step 3 (4 19) 
i----1 

hl(,,.Tx(c)(D)). 
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PLANAR CONVEX BODIES, 
FOURIER TRANSFORM, LATTICE POINTS, 
AND IRREGULARITIES OF DISTRIBUTION 

L. BRANDOLINI, A. IOSEVICH, AND G. TRAVAGLINI 

ABSTRACT. Let B be a convex body in the plane. The purpose of this paper is 
a systematic study of the geometric properties of the boundary of B, and the 
consequences of these properties for the distribution of lattice points in rotated 
and translated copies of pB (p being a large positive number), irregularities 
of distribution, and the spherical average decay of the Fourier transform of 
the characteristic function of B. The analysis makes use of two notions of 
"dimension" of a convex set. The first notion is defined in terres of the number 
of sides required to approximate a convex set by a polygon up to a certain 
degree of accuracy. The second is the fractal dimension of the image of the 
Gauss map of B. The results stated in terres of these quantities are essentially 
sharp and lead to a nearly complete description of the problems in question. 

1. INTRODUCTION 
Suppose B c N2 is a convex body: a convex compact set with nonenlpty interior. 
Many classical problems in analysis, geometry, and number theory are stated in 
terres of basic properties of such sets. For example, we may consider the difference 
between the number of lattice points inside the dilated set pB and its area, i.e., the 
discrepancy 
Dp(B) -- card (pB ç 77, 2) - p2 
where [-[ denotes the area. Among the many nat.ural questions we can ask about 
this problem (see the section on lattice points below) is, how does the geometry of 
B affect the growth rate of the discrepancy function? As we shall see, there are 
results that do hot distinguish among various convex sets. However, we shall also 
see that the behavior of the above discrepancy functions corresponding to different 
convex sets may vary dramatically, and that this behavior may be described in 
terres of natural and readily computable geometric quantities. 
The above question on lattice points bas a consequence in the study of irregu- 
larities of distribution. Suppose T' = {zj}jN=I is a distribution of N points in the 
unit square U = [0, 1] 2 treated as the torus 3F 2. Let B be a convex body in U with 
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dimneter smaller than 1. Assuine s _ 1, t E OE2. Then certain sharp upper estimates 
for the discrepancy 
N 
D(7), e, t) --=  ,'(eB-t(zj) -- IV « ]B I 
can be obtained from related estimates for lattice points (by a suitable trick we 
shall reduce to the case when N is a square, which in turn is a.n cas)- corollary). 
At the heart of the lattice point and the irregula.ritics of distribution problems is 
t.he Fouricr transfornl of the charaçteristic fimction of B. Out approach is to study 
the effect of the geometric propcrties of B on the dccav rate of the Fourier transform 
of thc charactcristic fimction of B and its variants. We shall then use this analysis 
t.o obtain precise information about thc discrepancy flmctions dcscribed above. 
How should we distinguish mnong thc various couvex planar sets? The lat- 
tice point problem mggests one natural approach. It was obscrved by Gauss that 
D,(B)  p, since thc bouudary of B is oue-dimensional. Cousider the case when B 
is a mfit square with si(les parallcl to the axes. When p is an integer, the boundary 
of pB contains  p integer lattice points, thus showing that this estimate cannot 
be improved. However, if B is a disc, the boundary of pB "curves away" from the 
integer lattice. Iu fact, it. is kuowu (see [161) that the estimate for Do(B ) in this 
case is much better. These two examples suggest that the curvature of the bound- 
ary may be the key distinguishing factor among convex sers. The boundary of the 
square bas no curvature, which leads to a poor discrepancy estimate, whereas the 
boundary of t.he disc has everywhere non-vanishing cnrvature, and the estimate for 
the discrepancy fimction is considerably better. 
The notion of curvature alluded to in the previous paragraph is the staudard 
geometric, or Gaussim, curvature, defined as the determinant of the differential of 
the Gauss map that maps each point on the boundary of a convex set to the unit 
normal at that point. It turns out that the geometric curvature alone does hot 
capture the relevant properties of convex planar sets fully. To see this, let us return 
to the case of the unit square. While it is true that the discrepancy function is 
terrible if the sides of the square are parallel to the axes. the discrepancy function 
becomes practically non-existent, even better than the discrepancy function for t he 
dise, if the square is rotated by a sufficiently irrational angle (see [14]). hl fact, it 
is precisely the "flatness" of a square that keeps its boundary froln hitting hardi)' 
any lattice points wheu it is rot.ated. This suggests that for "most" rotations, 
convex sets with "fiat" boundaries behave better as far as discrepancy functions 
are concerned. 
In this paper we consider the rotated and translated copies a -1 (pB) - t (where 
a ¢ SO(2), t ¢ "11 "2) of the dilated body pB (here p is a large positive nmnber), and 
we study the L 1 mean 

of the discrepancy 

 30(2) DP(c-I(B) - t)[ 

Dp(c-I(B) - t) = card ((pcr-l(B) - t) f3 Z 2) - p2 IBI. 
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The reason for choosing the L 1 mean alnong other L p means will be clear soon. Let 
us also say that iii many cases, averaging makes a discrepancy problem casier. For 
example, the Gauss circle problcln is a basic and unsolved problenl, while one tan 
obtain (sec e.g. [15] or [8]) a sharp result averaging in L 2 over translations of the 
discs and using only Parseval's ideutity and some properties of Bessel functions. 
Let us go back to the geonletry of B. The above observations tan be exploited 
in a number of ways. If "flatness" is good, then the family of rotated copies of B, 
is better if B is close to being a polygon. This lneans that B is good if it tan be 
approximated by a polygon with relatively few sides (the construction we are going 
to describe has been studicd iii [19] and [23], sec a]so [26]). We choose ai1 arbitrary 
point on the boundary of B and draw a chord to another point on the boundary of 
B in such a way that the maxinmnl distance from the chord to the boundary of B 
is p-. Roughly speaking, if the number of sides of the above inscribed polygon is 
< pa, we say that the dilnension of B is at least c (we shall explain later why for 
most of the paper we prefer not to consider the infinmln of the c's). Note that B 
is a polygon if and only if we can choose  = 0, and if B is a circle, then c = 1/2 
works. 
¥e can also take the following "(lual'" point of view. If B is close to a polygon, 
then its bomdary OB bas relatively few nornlals. A more precise way of saying this 
is that the area of the &neight)orhood of the image of OB under the Gauss map is 
< 5-d. If B is a disc, we can only take d = 1. On the other hand, we can choose 
d = 0 if and only if B is a polygon. As another example, let B be a polygon with 
infinitely many sides, the normals of which bave apertures in the sequence n -f, 
/3 > 0; it is easy to sec that in this case we can take d = (1 + j3) - 
Introducing the infima c* and d* (note that d* is the upper Minkowski dimension 
of the image of the Gauss map), we bave et* <_ d*/(d* + 1), and we can also prove 
that this bound is the best possible. Oll the other hand, we can show that c* can 
be as close to 0 as we want, even when d* is away froln 0. 
This paper is structured as follows. We shall first describe the nmin analytic 
idea, the effect of the geometry of a convex set on the average decay of the Fourier 
transform of the characteristic fimction of B. We shall also prove that polygons 
provide the fastest possible decay. We shall then apply out estimates to the dis- 
tribution of lattice points in convex domains and the problem of irregularities of 
distribution. 
In this paper most of the ideas used to prove the results on the average decay 
are new, while ahnost ail the applications to lattice points and irregularities of 
distribution are straightforward. 
We conclude the introduction by noting that a notion of a dilnension of a convex 
set nlay be applicable and natural in a number of interesting problenls in analysis 
and combinatorics. For exanlple, the Falconer distance conjecture says that if the 
Hausdorff dimension of a planar set is greater than 1, then the set of Euclidean 
distances among the points of this set has positive Lebesgue measure. However, if 
the Euclidean distance is replaced by the "taxi-cab" (11) nletric, the conjecture is 
clearly false, and in fact the set is required to have Hausdorff dimension 2 belote 
the sanie conclusion on the distance set is possible. It is reasonable to ask whether 
distances induced by convex sets with "intermediate dimension" provide examples 
of intermediate behavior in the Falconer distance problem. We hope to address this 
and other issues of this type in a subsequent paper. 
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1.1. L v average decay of the Fourier transform. The study of the decay of 
the Fourier transform 
as I1 --o oe is a classical subject. When 0/3 bas strictly positive curvature, then 
IB(ç)l  I1-3/2- However, when OB contains points where the Gaussian curvature 
nishes, then the above inequality is no longer truc. For example, when B is a 
polygon and  = (cos0. sin0), then B(pO) decays  p-] in me directions and 
 p- in most directions. In such ces it is usefifl to study the L p spherical average 
decay of B, given by 
where E is the unit. circle aud 1  p  oe. Here a bic result is Podkorytov's 
theorem 
(1.2) [[îB(P')][L(Z,)  p-3/2 
(sec [19]), where no regularity assumption on the boundary OB is required. 
Throughout this paper X  Y will mean that X  c)] with c depending only 
on the body B under consideration. MOl'eover, we shall always sume p  2. 
The study of (1.1) turns out to have applications to several problems, such 
 the distribution of lattice points in large convex domains ([20], [25], [7], [8]), 
irregularities of distributiou ([17], [7]), summation of multiple Fourier expansions 
([9], [5], [6]), and estimates for generalized Radon transforms ([21]). 
The paper [8] contains the following rather complete study of (1.1) under the 
additional sumption t.hat OB is piecewise smooth. When p = 2, (1.2) says that 
the rate of decay of (1.1) is independeut of the shape of B. When 2 < p  oe, any 
order of decay betwn that of the disc and of a polygon is possible. On the other 
hand, when 1  p < 2, a convex body with piecewise smooth boundary behaves 
either like a dise or like a polygon. In particular, when P is a polygon we have the 
sharp bound 
(1.3) [[P(P')[[L(a)  p-21°gp " 
and when B h piecewise smooth boundary, but is hot a polygon, we have the 
sharp bound 
(.) (p.)r¢)  cp -/. 
Actually, (1.4) is sharp whenever OB contaius at let one point where the Gaussian 
curvature exists and is different from zero. 
The above dichotomy, pointed out in [8], is no longer valid for arbitrary convex 
bodies. The existence of "chaotic" decays h been pointed out in [8, p. 553], 
using an abstrt argument on convex sets. Unfortunately, that argument is hot 
constructive, nor does it provide nontrivial explicit bounds for the average decay. 
The main analytic tool of this paper is the L p average decay for arbitrary convex 
planar bodies when 1  p  2. In essence, we shall consider the L  average decay 
and the L 2 average decay. The results for intermediate exponents tan be essentially 
obtained by interpolation. Roughly speaking, the L 2 average decay is an "all cats 
are grey in the dark" phenomenon, where the decay does hot distinguish among 
the different convex bodies. On the other hand, the L  average decay determines, 
in a sense, how close a convex set is to a polygon. 
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1.2. Inscribed polygons. We introduce the following notation. 
(cos 0, sin 0) and any small 5 > 0, let 
(1.5) h = max x-O, 
xEB 
T(B,5,0) = {e B: .o= 

For any 0 = 

We say that the chord r(B, 5, O) is of height 5, and we use it to define the following 
inscribed polygon (sec also [19] or [231). 
Definition 1. Let B be a convex planar body. Choose any chord of height 5, and 
name it ch1. Move counterclockwise constructing a finite sequence of consecutive 
chords of height 5 until you reach ch1. Then, if necessary, replace the last chord 
by one consecutive to ch1 (hence of height not greater than 5). In this way we get 
a polygon inscribed in B, and we denote it by Pfi. Of course Pfi depends on the 
choice of ch1, and we should write P(chl); however, none of our results depends 
on ch1, and, by a small abuse, we shall always speak about "the" inscribed polygon 
P. We denote by M be the number of sides of P. 
It has been proved in [23] that M < 5 -1/2. Our first result is the following. 
Theorem 2. Let B be a convex planar body and assume B_ 1  po (where 0 < 
c < 1/2, the cases c -- 0 and c = 1/2 being covered by (1.3) and (1.2) respectively). 
Then 

(1.6) 

Moreover, for any 0 < c < 1/2, there exists a convex planar body B such that 
Mo_, < pa and, for any ¢ > 0, 

limsup p-«+2+ II(p)ll,(,) > o. 
p-- -t- 

All the proofs will be given in the last section of the paper (§3). 
Before going on, we want to discuss the above theorem. The first step in the 
proof is to show that 

(see Definition 1). We are therefore reduced to estimating the average decay for 
a polygon with < p sides. The second step simply consists in recalling that the 
implicit constant in (1.3) depends on the number of sides of the polygon P, and 
that after reading the proofs in [7] or [8] one can rewrite (1.3) in the following way: 

0 2zr 
(1.7) [p(pO)[ dO <_ cNp -2 logp, 

where N is the number of sides of the polygon P, and the constant c is absolute 
(there is no loss of generality in assuming that the length of the boundary (OP is 
_< 1). Putting p in place of N, we then get (1.6). 
At this point one should expect to bave gotten a poor ïesult using the trivial 
estimate (1.7). The counterexample in the theorem shows that this is not so. 
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1.3. The image of the Gauss map. At every point of OB there is a left and a 
right tangent, therefore a left (-) and a right (+) outward normal. Let rr + : OB --* 
E1 be the map sending each point in OB to the leff/right normal. Also let 
(1.8) /B = -(0B) U +(0B). 
We identify Y]I with the iuterval [0.2rr). For every 0  [0.2r) we denote by al(O. A B) 
the distance between 0 and A B. For a given small 6, let 
(1.9) A = {z  [0.2rr)" d(x,A B) < 6} 

be the 6-neighborhood of A B. 

Theorem 3. Let 0 < d < 1. Assume 

(1.10) IAI 5 l-d. 
Then 

(1.11) 
M,,r«over, th«re «x'tsts a coin,ex body B satisfvmç IAfl  6t-d and su«h that 
limsupp ax+2+e ]I(P')]]'(z,) > 0 
p+ 
for any 
The proof will be given in the last section. 
Remark 4. Again, the ces d = 0 and d = 1 are covered by (1.3) and (1.2) 
respectively. 
R«mark 5. We point out that the infimum of the numbers d such that ]A]  l-d 
is just the upper Minkowski dimension of A, that is, the number 
30 
It is therefore possible to restate Theorem 3 in a form such  "If d > d*, then 
(1.11) holds'. However, we prefer to keep the original statement in Theorem 3 for 
the following two reons. First, the L.H.S. in (1.10) is the quantity that tually 
arises in the proof. Second, we do not want to confilSe naturally different objects, 
such  the polygons with finitely many sides and certain polygons with infinitely 
many sides (e.g. with an exponentially decreing sequence of slopes) which share 
d* = 0 with the polygons with filfitely nmny sides. For similar reons we did 
not introduce the infimum a* of the a's in Theorem 2. On the contrary, we shall 
introduce a* and d* in the following section in order to get a neater comparison. 
1.4. Comparing the previous arguments. For any B we denote by d* the 
Minkowski dimension of A (sec the above remark).  also denote by a* the 
infimum of the 's such that BÇ,  cp .  have the following theorem. 
Theorem 6. Let B be a convex planar body. Then 
d* 
- d*+l" 
Moreover, there exists B for which equality holds. 
The proof will be given in the last section. 
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Remark 7. Theorem 6 exhibits an upper bound for a* in terres of d*. A lower 
bound in terlns of d* does hot exist in general, since we can construct a family of 
convex bodies with the saine d* > 0 but a* arbitrarily close to 0. 
The proof will be given in the last section. 
The situation is different if we add geometric assunlptions ou B. 
Theorem 8. Suppose B is inscribed in a disc (i. e., B is the convex hull of a subset 
of a circle). Then * = d*/2. 
The proof will be given in the last section. 
The circle in the previous statelueut cau be replaccd by a closed convex smooth 
curve with everywhere positive Gaussian curvature. 
Remark 9. By appealiug to Theorem 2 and Theorem 6 we imlnediately get the 
following inequality, which is slightly weaker than the one in Theorem 3: 
1.5. A lower bound for ail convex bodies. The main results in this paper deal 
with "internmdiate" cases bctween polvgons and couvex bodies having a smooth 
convex arc in the boundary. These cases turn out be extreme. Iudeed, Podkorytov's 
theorem is a uniform (with respect to the choice of B) upper bound, while the 
following theorem gives a uniforln lower bound for the L  average decay of the 
Fourier transform. 
Theorem 10. Let B be a convex body in 2. Then 
limsup p21og-pHîB(p.)HL,(E,) > O. 
The proof will be given in the last section. 

2. APPLICATIONS 
2.1. Lattice points. Let B be a planar couvex body, and let er E SO(2) and 
t E OE2. We consider the discrepancy 
(2.1) 
where I'] denotes the area. The results in the previous section aud some arguments 
in [20], [25], [7], and [8] allow us to obtain several upper aud lower bounds for 
averages of the discrepancy (2.1) over rotations or rotations and translations. As a 
first example, it has beeu proved in [15], [25], and [7] that, for a polygon P, (1.3) 
implies 
fs Dp(a-l(P))l do- £ log2p  
0(2) 
As another example, one can use (1.2) to show that for auy convex planar body 
(Sec e.g. [15] or [8].) Note that (2.2) is false without the integration in t, as the 
case of a dise and Hardy's f-result (sec [16]) show. 
Again we lotus on the case p = 1, and we have the following result, which follows 
easily from Theorem 2 and some known arguments (sec e.g. [15], [25] or [7]). 
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Theorem 11. Let B be a planar convex body such that A/pç, < pa (0 < a < 1/2). 
Then 
2 0(2) 
Moreover, for every such a there exzsts a body 13 satis]ying 
limsupp-+  ID(a-(B)-t)ldadt>O, 
p-,+ 2 0(2) 
for any e > O. 
The proof will be given in the last section. 
R«mark 12. The cases a : 0 and a = 1/2 are known; see e.g. [71 and [SI respec- 
tively. 
2.2. Irregularities of distribution. Suppose T' = {Zj}jN__ 1 is a distribution of N 
points in the unit square U = [0, 1] 2 treated as the torus 2 "2. Let B be a convex 
body in U with diameter smaller than 1. Assume 
The study of the discrepancy 
N 
D(P,¢,a,t) = Z 
j=l 
has a long history (see e.g. the references in [2] and [17, ch. 6]). A typical result is 
the following theorem, due to Beck [1] and Montgomery [17, ch. 6] (see also [71). 
Theorem 13. Let B be a convex body in U = [0, 1] 2 with diameter smaller than 1. 
Then, for every distribution  = {zj }ï=l in U, 
fo o(2)f ID(T''¢'a't)ldtdad¢ > N1/4" 
The above result is sharp, since Beck and Chen [3] proved the following upper 
bound. 
Theorem 14. Let B be a convex body in U = [0, 1]  with diameter smaller than 
1. Then for every positive integer N there exists a distribution T' o] N points such 
that 
{ 1 } 1/2 
The above upper bound can be improved after replacing the L 2 norm with the 
L  notre. Indeed, Beck and Chen [4] proved the following result. 
Theorem 1. Let P be a convex polygon in U = [0, 1] 2 with diameter smaller than 
1. Then ]or every positive integer N there exists a distribution T' of N points such 
that 
(2.5) ID(T', ¢, a, t)l dt da de  log 2 N. 
0(2) 
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The next result follows easily from Theorem 11, [71 and [SI. The case a -- 0 
provides a different proof of (2.5). In the same way one can get a different proof of 
the L 2 result in (2.4) too. We point out that appealing to lattice point results does 
not work for L s norms when p > 2 and the body is a polygon (see [11]). 
Theorem 16. Let B be a convex body in U = [0, 1] 2 with diameter smaller than 1 
and such that MpB_ < p. Then for every positive integer N there exists a distri- 

bution 7 .) of N points satisfying 
fv2 Lo(2) 'D(T)'a't)' dadt < { 
where D(P, a, t) = D(P, 1, a, t). 
The proof will be given in the last section. 

log 2 N 
N +--d log N 
N/4 

when a = O, 
when 0 < a < 1/2, 
when a = 1/2, 

3. PROOFS 

The following known result (see e.g. [101, [19], [8]) will be used throughout the 
paper. 
Lemma 17. Let B be a convex body in 2. Using the notation in (1.5), we have 
I(pe)l 2 pli [i,.(B,p_l,e)l + ir(B,p_,,e + 7r)l ] , 
where }-} denotes the length of the chord. 

We define 
'(0, A B) = rein (d(0, AB), d(0 + 
and we deduce the following lemma. 
Lemma 18. For every 0  A B we have 
1 
Pro@ Let 0 ¢ A B (say 0 = -r/2). Assume that âB passes through the origin 
and that B lies in the upper half-plane. It follows that in a neighborhood of the 
origin âB is the graph of a nonnegative convex function, say y = ç(x), satisfying 
ç(0) = 0 and ç'(0-) < 0 < ç'(0+), where ç'(0-) and ç'(0+) denote the left and 
the right derivative at the origin respectively. Let 
E= {(x,y) E 2: y > ç'(0-)x and y > ç'(0+)x}. 
By convexity, B C E, and therefore 
1 1 2 
I(B,-l,e)l < + 
pç'(O+) plç'(O-)l pmin(ç'(O+),lç'(O-)l)" 
To complete the proof it is enough to observe that 
min (ç'(O+), }ç'(O-)l )  d(O,A B) 
and to apply the previous lemma. [] 

The following lemmas will be needed in the proof of Theorem 2. 
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Lemma 19. Let R >_ 1 and 0 < fl < r/4. Assume R/3 < 1/2. Denote by 
C = C(/3, R) the conv«x hull of the set 
{Re×p(iO)  - <  < Z} u {}, 
where the point P bas distance 1 from the points Re  and satisfies [P]  R. Then 
there exist posit{ve constants ci and c2 such that if Rp 2  ci then we bave 
[c(PO)[  c2R1/2p -3/2 
for every ]0]  /2. 
Proof. Integratiug by parts, we are reduced to estimating 
(3.) p- f ().  exp (2ip. x) d.. 
Jo 
C 
The boundary OC consists of two segments and an arc. In order to control the 
latter we reduce to the oscillatory integral 
exp ip j_lexp(ipRu) du 
) cl/2p-1/2 
for pR 2 large enough. The two segments have length 1, and their contribution in 
(.) i O (0-). 
Lemma 20. Let R > 1 and O < Ô < ¢/4. AssumeRa< . For any N  1, let 
B = B(, R, N) be the convex hull of the set 
{Rexp(2ikÔ/N),k = -N,...,N}  {P}, 
where, as belote, the point P has distance 1 from the points Re ' and satisfies 
]P]  R. Then there exist absolute constants c, c2, and c3 such that whenever 
p  2 and 
c c 2 N 2 
(3.2)   Rp   log2 , 
we have, for any -/2 
]B(p) OE c3R1/2p -3/2. 
Proof. Let C = C(, R) be as in Lemma 19. By (3.2) and Lemma 19 we bave 
]C(p)]  CR1/2p -3/2 
whe -/2  0  /2. 
We now study the Fourier transform ck- We claire that 
log N R 
(3.3) ]c(pO)]  «fl-1 N 
uniformly in 0. Indeed, CB is the union of 2N "lun" 1, .... 2N (each lune is a 
convex set bounded by a segment in B and by a portion of the arc in C, see Figure 
1) and, for any 0, 
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where f = f0 is defined by 
f(s) - 

Note that, for any given s, the above sure contains at most two terms. It is enough 
to consider one of them, i.e., we assume 0 <_ 0 <_ rr. Moreover, we are reduced to 
studying the case 0 _< 0 < CIN, the other cases being similar. In order to bound 
f(p), we estimate the total variation V I of the function f(s), which is thc length of 
the vertical segment in the kth lune. Now observe that 

whenever k _> 1 (see Figure 1). 

FIGURE 1 

Summing on k (there are N terms when 0 = 0 and N + 1 terres when 0 < 0 < 
ff/N), we get (3.3). 
Finally, for suitable choices of Cl and c2 in (3.2) we get 
I,,(pe)l _> Ic(pe)l- I\c(pe) I 
>_ C311/2 p_3/2 _ C4flp_ 1 log N R 
N 
 C5p--312R 1/2. [] 
Proof of Theorem 2. We start with the upper bounds in (1.6). Let PpS_l be as in 
Definition 1. Let /Bps_, be the smallest polygon having sides parallel to those of 
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ppB and containing B. It is hot ditïficult to see that for p sutïficiently large, 
where again the implicit constant depends only on B. By Lemma 17 we have 
I(,o)1 < p-' ç(,p-,o)[ 
nence, br th p« « («) i. []  
p- r(o-,cp-,O) eo  cM_p-21og(p)  cp-2+log(p), 
thereby proving (1.6). 
We now show that (1.6) is essentially sharp. Let B = B(, R, N) be  in Lemma 
20 and consider the sets B = B(, R, N), h = 1 2 3,..., where, for any small 
>0, 
R = 2 (-2),  = 2 (2--), N = 2 . 
We denote by  the union of the N sides and bv Ç the arc where they are 
inscribed. Observe that 
(.4)  Za < /4 
h:no 

for a suitable no. 

1 

FIGURE 2 

We recall that each Bh has the shape in Figure 2, i.e., it is a convex polygon 
consisting of two sides of length 1 and of Nh sides coming from a regular polygon 
of large radius Rb. Let Eh be the rotated and translated copy of every Bh, so 
that, moving counterclockwise, E, o = B, o and two consecutive Eh's have disjoint 
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interior and share a side (of length 1), while the union of the arcs Çh is a convex 
piecewise smooth curve. We write 
h 
By the condition (3.4), B is a convex set. Now let Ph = 2h. Let Ph 
Since (3.2) is satisfied, Lemnm 20 implies 
r,1/2 -3/2 = c2-h(«+l) 

or 
(3.6) 

1 2 
We then estimate the contribution of the convex sets Êh and Eh #, using Lemma 18. 
Indeed, since e satisfies (3.6), we obtain, for auy h, 
We then have 

_ 
J,+½ 

, r,1/2 -3/2 
I 
OE Clthlh tOh -- c2fl2l 
OE ]cl2h(a-«-2)--C2 2-2h1 
C 
>_ 3,  

To complete the proof we estinmte MpB_I. Given p _> 2, let H satisfy 2 H _( p < 2 H+I . 

Here we split 

H 2H a 
Observe that the first term is a polygon with -=no Ny  sides. Now consider 
that for any convex polygon Q and any 5, the number /I Q cannot exceed the 
number of sides of Q. Therefore the contribution of B« to Mpç, is  2 Ha = pa. 
As for Bb, we note that the length of [.J--H+ ÇJ is comparable to the length of 
ÇH, while the chords of height p- are longer, since [.J=H+ ÇJ cornes from flatter 
arcs. Therefore there are fewer chords than for ÇH. We have therefore proved that 
Proof of Theorem 3. Let çtp = A s In order to estimate 
p--1/(d+l)  
I(p) = Iîs(po)l de 

we write 

f 
f 
z(p) = / I(pO)l de + / I(pO)l de = h + X» 
J 
 J[o,2r]\2 
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To estimate I1 we use the Cauchy-Schwarz incquality, the fact that 
and (1.2)" 
2r I/2 
< p(a-I)/(2a+2)p-3/2 
In order to estimat.e I2 we tlse Lemma 18: 

(d+l) - logp 
(d+l) - iogp 
(al+l) - iogp 
 p-2  2k2-(t-d) 
k=0 
(al+l) - iogp 
= cp-e+. 
In order to give a counterexamp]e we use the body B constructed in the proof 
of Theorem 2. Again we consider the sets Bh = B(Bh, Rb, Nh), h = 1.2,..., where 
now 
Rb 2 h - - 
= '+, Zh = 2 h(-4, N = 2h, 
while Ph = 2 h- Arguing as in the proof of the previous theorem, we get, for every 
h, 
2-  + 
Ph Iîs(phe)ldO  c. 
JO 
To complete the proof it is enough to show that ]A2]  ,-d. We identi Aff with 
a subset of [0, /2], a»d we observe that 
consists of N points at distance /N. Given  > 0, we choose H so that 
<<  
N - N_ 

Hence 

We now split B = Ba U Bb as in (3.7). The contribution of Ba to 121  
j<_H 
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The llext proof follows ail argument hl [23]. 
Proof of Theorem 6. Let chi be a side of PB_I having elMpoints z and .q. Assume 
that, moving counterclockwise aloug the boundary of B, the point xd cornes before 
pi. Deuote by j the direction of the right normal in xj and by j the directiou of 
the left normal in pi. First observe that 
((3.8) follows by convexity when [ç -¢[ OE /4 and bv a trigonometric compu- 
tation when ] -¢[  /4.) Let  > a*. Summiug up and applying the HSlder 
inequality, we get 
3 
where the sure is on the fi@ sides of the polygon Pp-. It relnains to show that 
 ]ç -1 is bounded bv a constant independent of Pp-. Let 
Now observe that ifj  Z then the interval (çj,) C A Now choose d such 
thatd* <d< « Then 

k=O j_Zk 
÷çx 
2kg2 - - 
k=0 
« d 
=  2-'(- ) 
k=O 

so that card(Z) < 2 kd, and therefore 
1 - ¢1 -- _< 

The sharpness of the iuequality et* _<  follows frolll the COllllllOn counterex- 
ample in the proof of Theolem 2 a.nd Theoreln 3. [] 
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Proof of Remark 7. Let - > 1 and  > 0. For n _ 1, let xn -= n -f and yn -- n-f7- 
Let B denote the convex hull of the infinite points (x,,yn). We claim that the 
polygon Pv-' associated to B satisfies 

(hence a* <_ 1/73). Indeed, choose 

chl = B ç3 x, y) E 2 " y = P 
as the first side of Pv-. The number of sides of B located on the right of chl is 
. pl/,, and the claire follows since for aay polygon D with finitely many sides 
and any p we have .h/v°_, < # (sides of D). On the other hand, one checks that B 
satisfies 
and the exponent is the best possible (i.e., d* = 1/(3(- 1) ÷ 1)). 
If we now choose  = 1 + 1/3, we get d* = 1/2 and a* arbitrarily small (since 3 
can be large). [] 

Proof of Theorem 8. We show that a* = d*/2 whenever B is inscribed in a disc, 
namely when B is the convex hull of a subset of a circle. 
Let ppB_, be as in Definition 1 and assume a > a*, berce _h/vB_,  pa. Let 
x, x2,.., be the vertices of ppB_. See Figure 3. 

B 

! 
/ , 

/ 
/ 

FIGURE 3 

Let B1, B2,... be discs of radius p-12 centered at the above vertices. Since B is 
the convex hull of a subset of a given circle C, there exists a constant c such that, 
for any j, we are in at least one of the following two cases: 
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either 
i) cBj [.J cBj+I contains the arc iii EEB connecting xj and X+l, 
or 
ii) the part of OB connecting xj and xj+l and not contained in cB  cB+ s 
a segment. 
Indeed, sume that i) and il) fil. Then the arc in OB connecting x md +1 
must touch the unit c}rcle C outs}de of the d}scs cB) or cB)+l, at. a point hvin 
distance  p- from the side of , comecting  d +. Now observe tht 
this latter c be extended to a chord of C at distance  p-  from 0C. Them for 
a suitble c, the d}scs cB and cB+l cannot be d}st}nct. 
The above ilnplies that, for a 
Ap_/2 ç Cl 
kj=l 
and therefore 
op  
j=l 
hence, in this case, d*  2a*. 
We now prove that a* N d*/2. Let  < a*. Then there exists a sequence 
Pk  +oe such that M  p We clailn that there exist 
 k" k 
-/2 
that are  Pk sepated. Postponing for a nloment the proof of the claire, we 
conclude that 
which implies that the Minkowski dimension d* of A B cannot be snmller than 2, 
md therefore d* > 2*. 

Proof of the clazm. Let chi, pj and  be as in the proof of Theorem 6, and define 
It is enough to prove that whenever j  Sb we bave ç - @  Cfl[ 1/2. Since B is 
inscribed in a (unit) circle, a silnple geometric gument shows that. if 
p1/2, then the chord chi (which is a chord of B of height p) can be continued 
to a chord of the circle of height  fll and therefore of lenh  p/2. It follows 
that Ichjl £ çl/2, md (3.8) yields Içj - Fil  cP 1/2 for aily j = 1 ..... cp 

The following lemma will be needed in the proof of Theorem 10. The proof 
depends on as easy modification of an argument in [27]. 

Lemma 21. Let B be a convex planar body containing a large disc of radius r. 
Let g be a smooth nonnegative function supported in the set {t + v}teB ' Ivl< such 
that g(t) = 1 when t E B and dist(t,0B) >_ 1. Then there exists a constant c, 
independent of r, such that 
[[ff[[Ll() --> c log 2 r. 



3530 L. BRANDOLINI, A. IOSEVICH, AND G. TRAVAGLINI 
Pro@ We first need the following known inequality (see e.g. [24] or [13]). 
h E LI(IR) satisfy  e LI(IR) and (u) = 0 for u _< 0. Then 

Let 

(3.9) 

Ih(x)l dz 2 c - () du. 

A quick proof of (3.9) follows. Because of [12, p. 584] we can assume (u) _> 0. 
We then consider the odd real function s defined by s(x) = -i (1 - x)+ for x > 0, 
the Fourier transform of which is '(u) = (2rru - sin 2rru)/2rr2u 2. Then 

+ h(,) 
>_ c du. 
J1 

Observe that., through a translation, (3.9) implies the following fact. Suppose 
(u) = 1 for u in an interval of length r, say [q,q + ri, and that, moreover, (u) = 0 
foru<_q-1. Then 

(3.10) Ih(x)ldx > clogr. 
o 

To prove the lemma we may suppose that B lies in the hall-plane {(x, y)  x >_ 1} 
as in Figure 4. 

FIGURE 4 
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Then, by (3.9) and (3.10), 

= c log 2 r, 

since, because of the convexity of B, we can assume that g(x, y) takes value 1 inside 
a whole triangle such as the one iii the previous picture. [] 

Proof of Theorem 10. Arguiug by contradiction, we assume the existence of a pos- 
itive continuous function (p) - 0 (as p ---, +oe) such that 
(3.11) Ii"s (pO)l dO <_ e(p)p-2 logp 
for p _> 2. Let ç be a nonnegative radial cutoff function supported in the unit, dise; 
then the convolution 
g = XpB $ ç 
satisfies the assumptions in the previous lemma (pB contains a disc of radius  p). 
Therefore, by (3.11), 
log2p _< C 
= cP .L I'(pz)<(z)l dz 
< cP2 1 +  I(puO)l dodu 
= c 1 + p-ls IB(SO)I dOds 
( L + e(s) lOgs )s(l+ 
 c 1 + p_s)ds 

( Le(s) logs /+ /)lo) 
< c 1 + - ds + p s2 
 ,p 

= A(p). 
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To end the proof we observe that 

as p -, +oo, by l'H6pital's rule. 

A(p) 

log 2 p 

Remark 22. Using an induction argument as in [27], the above theorem can be 
extended to several variables, so that, for any convex body in IR ', 
limsup -=-1 [B(pcr)[ dcr > O. 
p-.+ log - p ,_ 
Remark 23. To prove our theorem we have used an idea introduced in [27] to get 
lower bounds for Lebesgue constants. Therefore otlr result shows a relation between 
the study of Lebesgue constants and the L 1 spherical averages of Fourier transforms 
of characteristic fimctions. However, we see no general theorem relating one to the 
other. See [18] for a related discussion with a number-theoretic flavor. 
R«mark 24. Estimating {r(B, 6, 0)1 (see (1.5)) is a geometrical problem which does 
hot necessarily involve the Fottrier transform. The previous theorem and the in- 
eqttality in Lemma 17 imply that, for any convex planar body, 
lim sup 1 
6--.0+ 61og(1/6) _ [r(B,6, O)[dO > O. 
The problem considered in the previous remark could be related to the study of 
floating bodies (sec e.g. [22]), where, in place of fixing 6, one fixes the area 
( 61r( B, EE, O)l) 
of the small part of B cut away by the chord r(B, 6, 6 0 i t.he direction O. 
Proof of Theorem 11. Arguing as in [15] or [7] and applying Theorem 2 and (1.2), 
we bave 
fv Lo() [Dp(a-(B) - t)] dadt 

0(2) O[m[p(l_2«)/(l+2a) 

I(ç)l 2 d« 

dr dt 
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< p2 

1/2 

The lower bound follows from Theorem 2 and the orthogonality argument in [7, 
p. 269]. [] 
Proof of Theorem 16. We prove only the case 0 < a < 1/2. Write N as a sum 
of four squares, N = j2 + k2 + 2 + m 2, and let al, a2, a3, a4 E [0, 1) be pairwise 
linearly independent on Z, so that, e.g., 
al+Pa2+ q 
for any choice of the integers p, q, j, k (j, k ? 0). That is, 
(3.12) (al + j-IZ) ¢3 ( a2 + k-'Z) = 0 
when j = k. Let 

and let us define Ak2, At, Am2 accordingly. Define 
P = Aj 12 Ak 12 At 12 Am. 
By (3.12) o has cardinality N. Since 
card (o ç B) - N 
= card (Aj ç B) - j2 ]BI +... + card (Am  B) - m 2 ]B[, 
it is enough to prove that, say, 
 £ I cara(Ai(«(B)+t))-j2.B.[dOdtNl°gN" 
 0(2) 
We can therefore prove the theorem suming N to be a square, say N = r 2, r  N, 
and 
Now observe that, writing w = (a, O) and applying Theorem 11, we have 
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= card (l)}p,q= 0 
 o(2)  
 o()  
= , L car'l (2  (ra(B) + u)) - r2 lBll duda 
0(2) 
 r 2/(+2) log r 
1 No/(+2o ) log N, 
where we have used tho fitct that, for a flmction J' L  (OE2) and for anv integer 
.¢0. 

.2 f(l,'u)du = f f(u)du. 

The above argulnent extends to several variables ail.er replacing the sunl of four 
squares bv Hilbert's theoreni (Vgaring's problem). 
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A FREE BOUNDARY PROBLEM FOR 
A SINGULAR SYSTEM OF DIFFERENTIAL EQUATIONS: 
AN APPLICATION TO A MODEL OF TUMOR GROWTH 

SHANGBIN CUl AND AVNER FRIEDMAN 

ABSTRACT. In this paper we consider a free boundary problem for a nonlinear 
system of two ordinary differential equations, one of which is singular at some 
points, including the initial point r ---- 0. Because of the singularity at r  0, 
the initial value problem bas a one-parameter family of solutions. We prove 
that there exists a unique solution to the free boundary problem. The proof 
of existence employs two "shooting" parameters. Analysis of the profiles of 
solutions of the initial value problem and tools such as comparison theorems 
and weak limits of solutions play an important foie in the proof. The system 
considered here is motivated by a model in tumor growth, but the methods 
developed should be applicable to more general systems. 

i. INTRODUCTION 

A special feature in tumor growth is proliferation; proliferating cells cause the 
tumor volume to vary in time, and, as a result, various models developed to de- 
scribe tumor growth are formulated as initial boundary value problems for partial 
differential equations with the tumor surface as a free boundary. 
It has long been recognized that a tumor contains different populations of cells, 
such as proliferating cells (i.e., cells that undergo abnormally fast mitosis), necrotic 
cells (i.e., cells that died because of lack of nutrition) and "in-between" quiescent 
cells (i.e., cells that are alive but their rate of mitosis is balanced by the rate of 
natural death). 
There are basically two kinds of models in the literature, according to which one 
of the following two assumptions is adopted: 
(i) different populations are segregated by interface boundaries; 
(il) different populations are mixed together in different concentrations. 
Models wth segregated populations wre developed by Greenspan [9], [10], Adam 
[1], Britton and Chaplain [2] and Byrne and Chaplain [3] (see also the references 
therein). In such models, necrotic cells occupy a central cote r < pi(t), proliferating 
cells occupy an outer layer p2(t) < r < R(t), and quiescent cells reside in the shell 
pi(t) < r < p2(t); in these regions nutrient and inhibitor concentrations satisfy 
reaction-diffusion equations. Rigorous analysis of models of this kind was given by 
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Friedman and l=(eiti«h [7] and Cui and Friedman [4] (for the case pi(t)= p2(t) = O) 
and by Cul and Friedlnan [5] (for the ceuqe p(t) = p2(t) > 0). 
Models of the type (il) were more recently developed in Ward and King [16], 
Pettct et al. [13] and Sherrat and Chaplain [14]. In this paper we are particularly 
interested in the model of Pettet et al. [13]. This model was developed in order to 
explain the experimental results of Dorie et al. [6], [7] with regard to internalization 
of particles injected into thc tumor across its surface. Previous models introduced 
in order to cxplain the saine experimental results were given bv McElwain and 
Pettct [12] and Thompson and Byrne [15]. 
As in other tmnor models, thc one dcveloped in [13] considers ail nutrients as a 
singlc spccies, and assumes that its concentration C satisfies thc diffusion equation 

(1.1) V2C = IC, 

wherc p is a positive constant, accounting for the consumption rate of nutrient 
divided by thc diffusion coeIficient. 
Another basic assmnption adopted by [13] is that dead cells are withdrawn im- 
mediately from the tumor upon thcir death, so that the tmnor contains only living 
colis. It is also assumed tha ail cells are incompressible and of the saine volmne, 
and that thc tumor is a sphcroid well packed with cells. It follows that the cell 
density within the tmuor is a constant, say, N. Thus, denoting by P and Q the 
densities of t he proliferating cells and quiescent oeils, respectively (i.e., the numbers 
of proliferationg cells and quicscent cells per mfit volume, respectively), we get 

(1.2) P+Q-= N. 

Next, proliferating cells are assmned to undergo mitosis at a rate KB(C) and 
become quiescent at a rate KQ(C); the quiescent cells are assumed to revert to 
proliferating cells at a rate Kp(C) and undergo necrosis at a rate No(C). This 
implies, by the law of conservation of mass, that 

OP 
(1.3) ôt 
OQ 
(1.4) 

-- + Ç " (ffpP) = (tçB(C) -- I(Q(C))P + Kp(C)Q, 

a-ï + V (ffQQ) = IX'Q(C)P- (IÇD(C) + tÇp(C))Q, 
where fie and ffQ are the velocities of proliferating cells and quiescent cells, respec- 
tively. The fim«tions A-B(C), IçD(C), Kp(C) and IçQ(C) are taken to be linear 
functions: 
t,'B(C) = BC. t,»(C) : »(Co - C). 
(1.5) 
/,-p(c) = pC, çQ(C) : Q(Co - c), 
where Co is a positive constant, and the coeflïcients kB, kD, kp, kQ are typically 
given by 
(1.6) kp = 0.05, k D =- 0.1, kB = 1, kQ = 0.9. 
The velocities ffp and ffQ are mutually related by the equation 
(1.7) ffQ = ffp + kV(", 
where ,( is a chemotactic sensitivity coefficient, assumed to be a nonnegative con- 
stant. The last assumption is based on some evidence that proliferating cells seem 
to be less motile as they undergo mitosis [11]. 
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(1.10) 
(1.11) 
(1.12) 
(1.la) 
(1.14) 
(1.15) 
where 
(1.16) 

Introducing the mean velocity 
1 (pffp + 
= we get from (1.3) aud (1.4) the equation 
1 {IçB(C)P -- l(D(C)Q}. 
(1.8) 7. 7 =  
Finally, the movement of the tumor surface is assule(l to i)e governed by the 
equation of continuity, nmnely, 
(1.9) dR. = 5'-ff, 
dt 
where R. = R.(t) is the radius of the tumor spheroid and ff represents the outward 
normal to the tumor surface. 
Equations (1.1)-(1.9) colnplemented by appropriate boundary and initial con- 
ditions form a model of evolution of a tulnor containiug only cells that are alive 
but in two different states (proliferating and quiescent) co-inhabiting the tumor. 
We are interested in (leveloping a rigorous mathelnatical treatment of this model. 
The present paper considers stationary solutious fol" the case ,k = 0; uon-stationary 
solutions and the more difftcult case ,( - 0 will be studied in future work. 
After rescaling space and rime, and settiug 
P Q c x 
p=-, q-N-l-p, c Co ff=Ulxl 
the stationary problem can be refornmlated in the following fonn: 
c +-c =pc, 0<r<R, 
ci(0) =0, c(R) = 1, 
2 
u' + -u= -Kv(c) + KM(c)p, O < r < R, 
(0) = 0, 
up'= tçv(«)+(IçM(«)--tçN(«))V--IçM(c)p . 0 < r < R, 
,(.) =0. 

K^I -- KB -t-/'D, ]£N ---- /£P -t- 
and KB(c), t(D(C), Kp(c) and KQ(c) are rescaled forms of the corresponding fllnc- 
tions given by (1.5), that is, 
tÇB(C) = CO'B c, tÇD(C) = C0],'D(1 -- c), 
(1.17) IÇp(c) = Co'pc, IÇQ(c) = Col¢Q(1-c). 
Indeed, equation (1.10) is the radially symmetric form of (1.1). The first bound- 
ary condition in (1.11) follows from the radial symmetry of the solution of the 
differential equation for c, and the second boundary condition means that the tu- 
mor receives suffmient nourishment from its host tissue (recall that IçQ (1) = 0 and 
IçD (1) = 0, which means t hat proliferating cells do not become quiescent and quies- 
cent oeils do hot undergo necrosis when the level of nourishment is c = 1). Equation 
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(1.12) is the radially symmetric version of (1.8) (with q replaced by 1 - p). If we 
substitute (1.12) into (1.3) and use the assumption that p is independent of t and 
X = 0, we obtain the equation (1.14). The boundary condition (1.13) is a conse- 
quence of radial symmetry of ff. Finally, the free boundary condition (1.15) is the 
stationary form of (1.9). 
The main result of this paper is the following: The stationary problem (1.10)- 
(1.16) has a unique solution (c,p, u, R) with the properties 
0<c(r)<l, c'(r)>O, 0<p(r)<l, p'(r)>0, u(r)<0 for 0<r<R. 
The lst inequality means that cells are moving into the interior of the tumor. 
Açtually, out analysis does hot depend on the linearity of the functions in (1.17), 
nor does it dcpend on the special form of the right-hand side of (1.10). In §2 we 
shall state our main result in a more general form and then outline the structure 
of this paper. 
We conclude this section by calling attention to the paper of Ward and King 
[16]. This paper introduces a modcl for a spheroid tumor with two populations 
of cells: proliferating cells and dead cells. The modcl is similar to that of Pettet 
et al. [13] with X = 0. However, the assumptions ruade .on the coeflïcients which 
appear in the equations that-are analogous to (1.3), (1.4) are quite different; thus, 
for example, the function IçB + IQ) in [13] is uniformly positive whereas in [16] it 
changes sign. In [16] the radius R(t) of the tumor is shown, numerically, to increase 
to infinity at a linear rate, so that one cannot expect to have stationary solutions. 

(2.1) 
(2.2) 
(2.3) 
(2.4) 
(2.5) 

2. THE MAIN RESULT 
Consider the following free boundary problen: 
c"() + 2c'() = F(c()), 0 <  < R, 
T 
c'(O) = O, c(R) = l, 
u'(r) + çu(r) = -IQ)(«(r)) + Iç^t(«(r))p(r), 0 < r < R, 
u(o) =o, 

u(r)p'(r)=Kp(c(r))+(KM(c(r))-K1v(c(r)))p(r)-KM(c(r))p2(r), 0 < r < R, 
(2.6) u(R) =0, 
with solution (c, u, p, R) satisfying 
ceC[O,n]C(O,n), ec[o,n]c(o,n), pec[onlc(o,n) 

(2.7) 
and 
(2.8) 

c(r)_>0, 0_<p(r)_<l for O<_r<_R. 

The functions on the right-hand side of (2.1), (2.3) and (2.5) satisfy the following 
assumptions: 

(2.9) 
(2.10) 

F(c), IçD(C), KM(c), KN(c) and Kp(c) are analytic in c, 0 < c < 1; 
F(O) = O, F'(c) > O for 0<c<l; 
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(2.11) 

KD(c)<O for 0_<c_< 1, and KD(1)=0; 
Koe(c) >0 for 0<c< 1, and Kp(O)--O; 
KM(c) = KB(c) + KD(C), where KB(c) satisfies the saine 
conditions as Kp(c) and K(c) + IçD(c) > 0 for 0 < c < 1; 
KN(C) = Ke(c) + KQ(c), where KQ(c) satisfies the saine 
conditions as KD (c). 

The conditions on Ks(c)and KD(C) imply that 

(2.12) K^(c)>O, Kz(c )>0 for O_<c_< 1. 

Note that the conditions in (2.11) are clearly satisfied for the fllnctions in (1.17); in 
particular, the inequality K(c)+ KD(c) > 0 follows from the assumption ks > kD. 
The main result of this paper is the following: 

Theorem 2.1. Under the assumptions (2.9)-(2.11), the .free boundary problem 
(2.1)-(2.6) bas a unique solution (c,p,u, R) with R > O, c analytic in [0, RI, p and 
u continuous in [0, RI and analytic in (0, RI, and 

(2.13) O < c(r) < 1, c'(r) > O if O < r < R, 
(2.14) 0<p(r)< 1, p'(r) > O if O < r < R, 
(2.15) p(R) = 1, p'(R) > O, 
(2.16) 

(2.19) 
(2.20) 
(2.21) 

c(o) > o, 
p(O) > o, 

u(r)<O i.fO<r<R. 
We shall use the shooting method to prove the existence result. The main idea 
is as follows: 
For each A E (0, 1) we denote by cA(r) the solution of (2.1) with initial value 
(2.17) cA(0) = A, c(0)=0. 
One can easily verify that cA(r) is analytic for r > 0, c'(r) > 0, 0«(r) > 0 for 
r > 0, cï(0) > 0, and there exists a unique finite number RA > 0 such that 
(2.18) cA(RA) = 1. 
Furthermore, RA <0for0<A< 1, RAoeifA0, andRA0ifA-* 1. 
We substitute c(r) = cA(r) into the system (2.3)-(2.5) to get 
u'(r) + u(r) = --KD(«A(r)) + KM(«A(r))p(r), 0 < r < RA, 
u(o) =0, 

u(r)p'(r) = Kv(cA(r)) + (KM(cA(r))- Kv(cA(r)))p(r)- KM(cA(r))p2(r), 
O<r<R» 

Hence the problem of solving the system (2.1)-(2.6) is t ransformed into the following 
problem: Find values of A and corresponding solutions of (2.19)-(2.21) such that 
the free boundary condition 

(2.22) u(RA) = 0 

is satisfied. 
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Due to the singularity atr = 0 of the differential equatiols (2.19), (2.21). the 
shooting lnethod we use iii this paper turns out to be quite diffelent from the 
standard one, lmleed, we shall prove that, mdike other nonsingular ODE problems. 
solutions of the initial value probleln (IVP) (2.19)-(2.21) exihibit the following 
iltel'esting l»hellomenol: Tlmle exists a çritical vahle A  (0, 1) such that, for 
anv A < A < 1, the IVP bas a. torique solution, while for each 0 < A < x. the 
IVP has a. ctmtilmUln of solut.iols (po, tt,) (  R).  shall also prove that the 
value of A for which the free boundary prol»leln (FBP) (2.19)-(2.22) h a solution 
must belong to (0, A). Thus. out "shooting target'" h to be reached in two 
steps: First, for everv   (0, .) we lmed to find a   R su«h that the sohltion 
(p,t)  (P,x&-u&) of the IVP possesses tiw "best" approxilnate properties to 
the solution of the F'BP. Having detel'milmd this special solutiol (p. u.) for each 
 G ((), .), we thon pro«e,d with the second t«'p of dctermining a value of  for 
whi«h (p, u) satisfies tbe frec boundary comlitiou (2.22). 
Thc structure of the pal»er is as follows: 
In 3 we introduce some auxi[iarv functions of  and studv their basic properties. 
These flmctions play a flmda.menta[ rolc throughout this papcr. 
bi order to prove Theor,m 2.1 we first a'ed to solve the IVP near r = 0. Since 
the equa.t.ions (2.19) ami (2.2l) (particularly thc second one) are singular at r = 0. 
local existence of solutions of thc IVP is hot cnsured by classical results. Sectons 
4 and 5 are devotcd to establishing local existence (i.e., for 0 < r < 6, 6 small) 
of solutions for the IVP. In 4 we consider only analytic solutions, k prove that 
there exists a scquece {A,,}nt convcrging to A increasingly, such that for each 
A ¢ An, (2.19)-(2.21) has a unique analvfic solution, while for each An, the svstem 
bas either no a.nalvtic solutions or a. continuum of analvtic solutions. 
In 5 we consider the IVP for non-a.nalytic solutions. The key step is to transform 
the IVP into an equivalent system of integral equa.tions that can be solved by 
using the contraction mapping principle. The final result is as follows: for each 
A < A < 1, there is a unique solution, whereas for each 0 < A < AOe, there 
exists a contimmm of clsical solutions (p(r; A. ), u(r; A, )) depending on a real 
parameter . 
The family of solutions (p(r; A.), u('r A,)) does hot depend continuouslv on 
(A,') at the points A = A. To overcome this difficultv, we introduce, in 6, a 
parmneterization  = (, ) with a new parameter  G R such that the solutions 
will depend contimmusly on (A,) for ail A G (0, A) and  G R (for 0 5 r < 6, 
for some 6 > 0). This solution will be denoted by (p, t¢,). 
The next step is to studv the profiles of p,(r) and u.(r) and use this informa- 
tion to extend the solution to either the entire interva.l [0, R] or a mximal interval 
[0, ) such tha.t p¢(r) blows up at r = . This is done in 7, where we shall prove, 
in pa.rticular, that p,(r) can change monotonicity (from increing to decresing) 
at most once. It will also be clea.r from the discussion of this section that the values 
of A for which the free boundary condition (2.22) can be sa.tisfied must belong to 
(0, ). 
In 8 we derive another integral equation fornmlation of the IVP and us it to 
introduce the concept of weak solutions. Unlike the integral equation fornmlation 
introduced in 5, which holds onlv locally, the integral equation formulation derived 
in this section holds globally. It will enable us to work with weak limits of solutions. 
The main result of this section asserts that a. weak solution (p. u) of the IVP is 
a.ct.uallv a cla.ssical solution provided p  0. 
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In §9 we shall perform the first step of the shooting method; that is, we shall 
prove that for any A  (0, Aoe) there is a unique  such that (p, u) = (p6, ) 
is either a "subsolution" or a "supersolution" of the FBP. By a subsolution of the 
FBP we mean a solution (p(r), u(r)) of the IVP, defined for all 0 _< r _< R, that 
satisfies the conditions 
O<p(r)< 1, p'(r)>O for 0<r<R, 
u(r) < O for O < r < ro, u(ro) =O, u(r) > O for r0<r_<R, 
for some 0 < r0 _< R. By a supersolution of the FBP we mean a solution 
(p(r), u(r)), defined also for all 0 _< r _< R, that satisfies the conditions 
O < p(r) < 1, p(r) > O, u(r) < O for 0<r<R, 
and p(R)) = 1, u(R)) < 0. The discussion of this section shows that subsolutions 
and supersolutions possess the "best" approximate properties to the solution of the 
FBP among ail solutions of the IVP. We shall also prove that a unique subsolution 
exists for each A near A. 
In §10 we shall perform the second step of the shooting nlethod to get a solution 
of the Ft3P. First, we prove that the three sets 
B0 = {A  (0, A)  A corresponds to a solution of the FBP}, 
B : {A  (0, A)  A corresponds to a subsolution, 
but not a solution of the FBP}, 
B2 = {A  (0, Aoe)  A corresponds to a supersolution of the FBP} 
do not interscct each other and their union is equal to (0, Aoe). Next, we prove that 
BI and /2 are open sets. Finally, we show that every A near 0 belongs to Bee, so 
that B2 ¢ . Since also BI ¢  (by §9), we conclude that B1 t2 B2 ¢ (0, A), 
so that B0 ¢ , which means that there exists at least one solution of the free 
boundary problem. Uniqueness is proved in §11. 
Some fornmlas used in §5 and §6 are proved in the Appendix (§12). 
To end this section we want to emphasize that throughout this paper we consider 
only solutions that satisfy the condition p(0) 2 0; this is of course motivated by 
the fact that p(r) represents the density of cells. The condition p(0) 2 0 appears 
implicitly in several places in this paper. 

3. AUXILIARY FUNCTIONS 
In this section we introduce several functions of A that will play a fundamental 
role throughout this paper. 
Lemma 3.1. For any 0 < A <_ 1 the quadratic equation 
(3.1) Kp(A) + (KM(A) - h'v(A))c - h',(A)a 2 = 0 
bas a unique positive solution, which we denote by a(A), and 
0<a(A)< 1, a'(A)>0 for 0<A< 1, a(1)=l, 
{ 0 if I(D(O )  
(0+) = (/,»(0)-/,Q(0))//«»(0) if «»(0) > ,'Q(0). 
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Pro@ Denote by G(, A) the left-hand side of the equation (3.1). It is clear that 
forall0<A<l, 
(1, A) = K»(A)-/«(A) = -K(A) < 0, 
G(0, A) = Kp(A) > 0, lim G(a,A) 
Hence the equation (3.1) has exactly one positive root in the interval (0, 1). It is 
given by 
(3.2) 
1 
a(A) -- 2Km(A) (KM(A) - Kv(A) + x/(KM(A) - Kv(A)) 2 + 4KM(A)Kp(A)). 
By (2.11) and (2.12), it follows that 
oa(, A) = K(A) + (K,(A) - 
(3.3) 
= K(A)(1- )- h'(A)( + K,(A)a(1- a) > 0 
for0<a< 1,0<A<l, and, by(3.2), 
OG 
(3.4) 
= - \/(/«,(A) - K(A))2 + 4K^(A)'»(A) < 0 
for 0 < A < 1, so that 
(3.5) '(A) - OG /OG 
oA ((A),A) b-((A),A) > o 
for 0 < A < 1. The rest follows immediately from (3.2). [] 
By the above lemma, it is reasonable to define (0) = a(0+). From (3.2) one 
easily finds that a(A) is also differentiable at A = 0, 1, and m'(0) > 0, s'(l) > 0. 
We introduce the function 
1 
(3.6) ff(A) = 5(- h'D(A) + Kt(A)(A)), 0 _< A <_ 1. 
Lemma 3.2. We bave fl'( A) > 0 for ail 0 <_ A <_ 1, and there exists a unique 
number Aoe E (0, 1) such that 
{ <0 for 0_<A<A, 
(3.7) (A) = 0 for A = 
>0 for A<A<_I. 
Pro@ Since KM(A) = KB(A) + Km(A) and K(A) > 0, K)(A) < 0, we have 
1 
fl'(A) = g(- K)(A)(1- + K(A)a(A)+ Km(A)a'(A)) > 0 
for 0 _< A _< 1. Next we note that 
ff(0) = --½KD(0)(1 -- (0)) < 0 and ff(l) = ½K.(1) > 0. 
Hence there exists a unique A = A such that (3.7) holds. [] 
For every integer n OE 0 we introduce the function 
-ï,(A) = n/3(A)- (KM(A)- KN(A)) + 2K»t(A)a(A) 
(3.8) 
= nf(A) + v/(KM(A) - Kv(A)) 2 + 4KM(A)Kp(A), 

where 0_< A < 1. 
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Lemma 3.3. (1) If A _ A _ 1, then ïnA) > O for all n, and if O 
then 
71() > 72(A) > .-- > 7n() > 7n+1() > .--, 
(3.9) lira 7n(A)=-oe. 
n- (: 
(2) There exists a positive integer no such that for every n >_ no the followzng 
assertions hold: 
(a) 7'(A) > 0 forO <_ A <_ 1, 
(b) the equation 7n(A) = 0 has a unique positive root An, and 
(e) the sequenee {n}n%no iS nonotone in«easing and 
(3.10) lira An = . 
n-- 
(3) For 1 <_ n < no, the set of zeros of ïn(A) is either finite or empty. 
Pro@ Assertion (1) follows immediately from Lemma 3.2. Next we note that ï'() 
is the sure of n/Y() and a bounded contimlous flmction. Since 
(3.11) /'(A) _> const. > 0 
for 0 <_ A <_ 1, it follows that (a) holds for sufficiently large n. Thus, for large n, 
7n(A) cannot bave more than one zero. Since 
7n() = x/(/«^l() -/«N()) 2 + 4/«^()/«(A) > o,  = 1,2,..., 
and, for large n, 
n/( 
7n(0) = - "D(0)(1 -a(0)) + [KD(0)- KQ(0)[ < 0, 
we sec that (b) holds also for sufiïciently large n. The assertion that {An}n=no is 
monotone increasing follows from (3.9) and the fact that 7'() > 0 for large n. To 
prove (3.10) we note, by (3.8), that 
1 
('n) = ---- V/( Ix'M('n) -- /X'N('n)) 2 ÷ 4I'M("n)I'v("n ) -- 0 
as n  oe. Since (A) = 0 and '(A) > const. > 0 for 0 < A < 1, the assertion 
(3.10) follows. Finally, the assertion (3) follows from the fact that 7n(A) is analytic 
inAforall0<A_< 1. [] 

4. ANALYTIC SOLUTIONS OF THE INITIAL VALUE PROBLEM 
In order to solve the free boundary problem (2.19)-(2.22), we first need to inves- 
tigate general solutions of the initial value problem (2.19)-(2.21). Since, by (2.20), 
the equation (2.21) is singular at r = 0, the existence of solutions of this problem 
does not follow from a standard theory. To get an insight into the construction of 
the general solution, we begin by considering solutions that are analytic near r = 0. 
The main result of this section is the following: 
Theorem 4.1. Let 
S= {A  (O, 1) " ïn(A)=O forsomeintegern >_ 1}. 
Then, for any A  (0, 1)\S, there exists a unique analytic solution of (2.19)-(2.21) 
in some interval 0 _ r _ 5, 5 > O. If A  S, then the system (2.19)-(2.21) either 
has no analytic solutions, or it has a 1-parameter family of analytic solutions. 
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Proof. Since F(c) is analytic in c for 0 < c < 1, cA (r) is analytic in r for 0 < r < 1 
and 0 < A < 1. We shall prove that for A E (0, 1)\S the analytic solution of 
(2.19)-(2.21) is given by the power series 

 r n  r n (n)to.  
(4.1) u(r) = Z . uln)(0)' p(r)= Z .P t , 
n=O n=O 

where the derivatives ul n) (0), pi n) (0) are computed inductively from (2.19)-(2.21), 
and they satisfy the inequalities 

HoH n-i HoHn-ln! 
(4.2) I.(ï)(°)l < 2 !, IP(n)(°)l-< 2 

for n _> 1 and some positive constants Ho, H (depending on A); here Ho is such that 
(4.2) holds if n = 1, and H will be specified later on. Clearly, if these assertions 
are proved, then the first part of the theorem follows. 
We begin by forlnally computing the dcrivatives u(ï)(0), p(ï)(0). By (2.20) we 
bave u(0) = 0, so that, by (2.21), 

Kp(c(O)) + (KM(cA(0)) - KN(c(O)))pA(O) - KM(cA(O))Pî(O) = O. 

It follows, by Lemma 3.1, that 

(4.3) 

pA(0) = c(cA(0)) = c(A), 

and then, from (2.21), 

(4.4) 

,i(0) = ½(- K()+ K^,().()) = Z(). 

Next we differentiate the equation (2.21) and take r = 0. Since c(0) = 0, we get 

(4.5) %(0)p(0) 

and since 71(A) ¢ O, 

(4.6) p(O) = O. 

Suppose we have colnputed ulm)(0), plm)(o) for m = 0, 1,--- ,n- 1. Multiplying 
(2.19) by r, differentiating n times and taking r = 0, we get 

Tt k -1)(A) n t- -- 
(4.7) u )(0)-- n+2 n+2 

n--1 
Z (n-lTft) 
m0 

where 

(4.8) km)(A) d'n] 
-- h'(cA(r)), i= DM,N,P, re=O, 1,-.-. 
drm r=O 
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Similarly, by differentiating (2.21) n times and taking r = 0, we obtain 
m=l 
m l )n M ta)P tu)P (0) 
(4.9) = =o 
n 
m )u tu)p (0). 
Since   S, there exists a constant co > 0 (depending on ) such that 

I(A)I _> c0 
for all n >_ 1. Since c(r) is analytic in r and K(c) is avmlytic in c, i = D, M, N, P, 
it follows that Ki(c(r)) is also analytic in r, which implies that 
A°A"m! (i= D,M,N,P, re=l,2,---) 
for some constants A0, A (depending on A). Using these inequalitics, one can now 
eily prove that if (4.2) holds for all 1  n  m, then 
CH°H- (m + 1), 
CH°H-(m+ I) ' Ip+)(0)l< (re+l)2 
I+')(°)l  ( + 1) - 
where C depends only on A0, A and H0. Taking H  C, we conclude, by induction, 
that (4.2) holds for all n  1. 
Suppose next that  G S. Then there exists a positive integer n such that 
7n() = 0. It follows that (2.19)-(2.21) cannot have an aualytic solution if the 
right-hand side of (4.9) is hot equal to zero. If instead the right-hand side of (4.9) 
vanishes, then we can take an arbitrary value for pn)(O) and argue similarly  
above to conclude that (2.19)-(2.21) h an analytic solution for eh choice of 
£")(0). 

Remark 4.1. If  E S, then both of the two cases mentioned in Theorem 4.1 can 
occur. For instance, by (4.5) we see that if "h() = 0, then p(0) can be any 
real number, so that the system (2.19)-(2.21) has a 1-parameter family of analytic 
solutions. On the other hand, for n = 2 the equation (4.9) reads as follows: 
(4.10) 
2()p(0) = (K() + (K,()-/«())()-/«/()2())«ï(0) 
---(2K^/(A)c(A)- (KM(A)- KN(A)))c'(A)«ï(0) (by (3.3)-(3.5)) 
:V/(KM(),) -- KN(A)) 2 + 4K^l(A)Kp(A)c'(A)cï(O) (b  (3.2)). 
Since the right-hand side is positive for all A E (0, 1), the system (2.19)-(2.21) 
cannot have analytic solutions if 2 (A) = 0. 
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The above examples show that analytic solutions do not depend contmuously 
on A. Therefore, by working with analytic solutions we cannot use the shooting 
method to find a solution of the free boundary problem (2.19)-(2.22). In the next 
section we shall consider general (non-analytic) solutions of (2.19)-(2.21) in a small 
interval 0 _< r _< 5. 

5. NON-ANALYTIÇ SOLUTIONS OF THE INITIAL VALUE PROBLEM 
In this section we shall consider general classical solutions of the problem (2.19)- 
(2.21). By a classical solution of (2.19)-(2.21) on an interval [0, 5] (5 > 0) we mean 
a pair of finctions p, u in CIO, 5] ç) C(0, 5) that satisfy the equations (2.19), (2.21) 
for 0 < r < 6, with u(0) - 0. 
We first consider solutions of (2.19)-(2.21) in the class 
(5.1) u Ç C'[0,51, p Ç C'[O, 5] 
for some 5 > O. By (2.21) and (2.19), for sueh solutions we have 
(5.2) p(0) = (), '(0) = #(). 
We shall also impose the condition 
(5.3) p'(O) =0. 
This condition is satisfied if p Ç C2[0, 6] and "1() # 0 (but see Theorem 5.4). 

We introduce new variables P, U by 
(5.4) p(r) = () + rP(), 
Then 
(5.5) P Ç CIO, 5] ciel(0,6], 
(5.6) P(0) =0, 
and 
(5.7) 
(5.8) 
where 
(5.9) 
(5.10) 
(5.11) 
fA(r, P, U)= (-- [ç-M(cA (r ) )p2- 
() r 
where 
(5.12) 
(5.13) 

() = (() + u()). 

U Ç CIO, (] Ç1 ci(o, (], 
U(O) =0, 

U'(r) + U(r) = KM(c,(r))P(r) + m,(r), 
P'(r) a(A) P(r) = f,(r,P(r),U(r)), 
r 

"ïl()) 
«() - 
()' 

1 
.,(,-) = (-/:,(c,(v)) + tc(c,(v)),(,)- 3Z()), 

g(O) UP g(r)-g(O) p+ 

r 

h() 

9,(r) = 2K^t(c,(r))a($)- (K^t(c.(r))- KN(C.(r))), 
hA(,') = I<v(c(,-)) + (/<(c()) - tcN(c(r)))() - t,',(c())() 
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Since the function --KD(cA(r)) + KM(cA(r))a(A) -- 3/(A) is in C  up to r -- 0 and 
is equal to zero at r -- 0, the function mA(r) is in C  up to r -- 0. Similarly, the 
functions (gA(r) -gA(O))/r and hA(r)/r 2 are also in C  up to r -- 0. Furthermore, 

(5.14) 

(5.15) 

(5.16) 

hA(r) 1 ,, r , 

+ (/«i()- K}()).()- Kb().2()} + 0(). 

We shall later on use the inequalities 

a(A)>-I if A(0, A); a(A)<-I if A (A,I). 

We want to recast the differential equations for U, P as integral equations. 
Clearly, equation (5.7) with the condition U(0) = 0 is equivalent to the integral 
equation 
(5.17) U(r) = - KM(cA(p))P(p)p3dp+ - mA(p)padp. 
In order to derive an integral equation for P, we first introduce some new notation. 
For any A  [0, 1]\S we define inductively (,,(A) (n _> 0) and fl,,(A) (n _> 1) as 
follows: 

(5.18) 
(5.19) 

(5.20) 

o()=(A), (A)=(A), 

for n = 1,2,---, where k} '0 (A) are as in (4.8). Note that 
.() = p")(o), .() = ,")(o) (eo() = o), 
for 0 < A < 1, where (pA, uA) is the unique analytic solution of (2.19)-(2.21) (cf. 
(4.7), (4.9)). Note also, by (5.3) and the fact that c(0) - 0, that 
() = 0, /() = 0. 

We also define 
,,+()7,,+() ,,+() 
(5.21) #,,(A) = = 
( + 2)!() ( + 2)! 

(n + 1 - a(A)) 

(n --- 0, 1,2,---). 
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For A E S, "h(A) ¢ 0, if n is the nonnegative integer such that 7n+2(A) ---- 0 
(i.e., o-(A) = n + 1), then o(A) (0 <_ i _< n + 1), /3(A) (1 <_ i _< n + 2) and #(A) 
(0 < i < n - 1) are still well-defined by (5.19)-(5.21). We also define #,(A) by 

m=2 

This is consistent with (5.21), since 7n+2(A) appears as a denominator in the defi- 
nition of an+2(A) (see (5.19)). 
The following lemlna gives an equivalent integral form of the equation (5.8). 

Lemma 5.1. Let P(r), U(r)  C[O, SlfC'(O,5 ] be a solutzon of (5.6)-(5.8) and 
assume that A  Aoe. Then. if 5 is sufljîciently small, the following assertions hold: 
(1) /f «ith«r p(A) > 0 or 7(A) _< 0 (<=:îz a(A) _< 0), thon (P(r),U(r)) satisfi«s 
the equation 

(5.23) P(r) = r «() f(p,P(p),U(p))p-«()dp. 
(2) If7(A) > 0 and 72() < 0 (<=:= 0 < a(A) < 1), then the limit 

 = lira r-«()P(rl''. 
r--O 

exists, and ( P(r), U(r) ) satisfies the equation 

0 r 
(5.24) P(r) =wr «(A) + r «(A) f(v,P(v),U(v))V-«(A)dv. 
(3) If 72() = 0 (<=:îz a(A) = 1), thon th« limit 
w = lira r-(P(r) - #o(A)r logr) 
r---O 

exzsts, and ( P(r), U(r) ) satisfies the equation 

(5.25) 

P(r) =wr + #o(A)r log r + r (f(p,P(p),U(p))- #o(A))p-dp. 

(4) In general, if for some integer n >_ 2 we bave EE(A) > O. EE+(A) < 0 
(,îz n - 1 < a( A ) < n), then the limit 

n-1 
 = lira r -«('x) P(r) - 
--.o (m + 1)! 
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exists, and (P(r), U(r) ) satisfies the equation 

(5.26) 

n-1 
p() = «a) +  'm+l().. 
=1 (m + 1)! 
r n--2 
grt---0 

If instead %+1() = 0 (=:: a($) = n), then the limit 

n--1 
02-----lilI, .-n(p(.)_ E °m+l(/) rm -- P,-l(A)rn log r) 
,o ç (m + 1)! 

exists, and (P(r), U(r) ) satisfies the equation 

(5.27) 

n-1 
P(r) = wr n + E am+l())rm + P'-l(A)rnl°gr 
m=a (m + 1)! 
r n-2 
+.'1. (:(.,.(.),.(.))- 

Conversely, il P(r), U(r)  CIo, 5] and they satisfy the coupled system of equations 
(5.17) and one of the equations (5.23)-(5.27) (in accordance with the correspondmg 
condition on A), then (P(r), U(r)) is a solution of the problem (5.5)-(5.8). 

We need the following lennna: 

Lemma 5.2. (1) Let P(r), U(r)  C[O, 5] and suppose that they satisfy the equation 
(5.17). Suppose further that, for some integer n >_ 1, 

(5.28) PO) =  m()m-l! 
+ O(r ') as r  O 
m- l 

Then 

(5.29) 

u() = /m(____))m-1 + o(.+) a   O. 

(2) Let f(r,P(r),U(r)) be the function 9iven by (5.11) and A ¢ Aoe. I P(r), 
0") a t ,nion (5.28) ,nd (5.29), tn 

(5.30) A(r.P(r),U(r)) =- E Pm(A)rm + O(r') as r  O. 

Pro@ Since the functions KM(c(r)) and m(r) are in C  up to r = O, we have 

n 

m=O 

+ O(r'+l), 

as r --, 0, and a similar expansion holds for ma(r). Substituting these expressions 
and (5.28) into (5.17) yields (5.29). Next, since (ça(r)-ga(O))/r and ha(r)/r  are 
also in C  up to r = 0 and U(0) +/3(A) ¢ 0, it is clear that fa(r,P(r),g(r)) bas 
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an expansion 

n--1 
f(r,P(r),U(r)) =  a,(A)r " + O(r ') as r - O. 
n-O 
The proof that ara(A) = #,() is given in the Appendix (§12). 

Proof of Lemma 5.1. Consider Case (1) first. It is clear that fA(r, P(r), U(r)) is 
bounded for r  (0, 6] if 5 is sufficiently small. Thus multiplying (5.8) by r 
and integrating over the arbitrary interval [0, ri immediately yields the equation 
(5.23). 
Consider Case (2) next. Since P(r)  C[0, 5], there exists a constant C (depend- 
ing on  and 6) such that 

(5.31) IP(r)l <_ C for 0 _< r _< & 

In the sequcl we shall use C to denote various positive constants (depending on A 
and 6). By (5.17) it follows that 

(5.32) Iu(r)l _< cr 
Thus U(r)/r is bounded, and 

for 

[U(r)+3(A)I>C>O for O<r<5 

if 5 is sufficiently small. Consequently, 

(5.33) IfA(r,P(r),U(r)) I <_ C for 0 < r _< 5 
and, since 0 < er(A) < 1, fA(r, P(r),U(r))r -«() is integrable on (0,6]. We now 
multiply (5.8) by r -«() and integrate over the interval [s, ri, for arbitrary 0 < s < 
r _< 6, to get 
(5.34) r-«()P(r)- s-«()P(s) = f(p,P(p),U(p))p-«()dp. 
Since the right-hand side converges to zero as r - 0, the limit 
w = lira r-«(A)P(r) 
r--*0 

exists. Multiplying (5.34) by r -«() and letting s - 0, (5.24) follows. 
Consider Case (3). We rewrite (5.34) in the form 
(5.35) 
r-lp(r)-s-lP(s)= (f(p,P(p),U(p))-#o(A))p-ldp+#o(A)(logr-logs) 
We claire that 
1 
(5.36) IP(r)l <_ Crlog - for 0 < r _< 6. 
Indeed, as before, (5.31)-(5.34) hold (with a(A) = 1), so that 
15-1P(5) - r-P(r)l _< Ih(p,P(p),U(p))lp-ldp <_ Ç(log5 - logr) 
for 0 < r _< 6, and (5.36) follows. Substituting (5.36) into (5.17), we find that 
1 
(5.37) IU(r)l _< er 2 log- for 0 < r _< 6. 
T 
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It follows that 
If(r,P(r),U(r)) - o(A)l 
(5.38) <-C[P(r)IU+CI U(r (r) l÷CrIP(r)l÷lru(U(r)÷/3(A)) 
<_ Cr u log u r + Cr u log u r + Cr u log 1 
- + Cr < Cr, 
r 
so that f: If (P, P(P), U(p))- #0(A)[p-dp < oo. Consequently, by (5.35), the limit 
w = lim r -1 (P(r) - #o(A)r log r) 
r---O 
exists, and letting s --, 0 in (5.35) we see that (5.25) holds. 
Consider now the general case'n(A) > 0,-n+l(A) < 0, namely, n-1 < er(A) < n, 
where n _> 2. As before, (5.31)-(5.34) hold, so that 
5 
]5-«()p(5) _ r-«()p(r)] < .£ I f(p,P(p)'U(p))]p-«()dp 
_ C(51-«(A) _ rl-«(A)) 
for 0 < r < 5. It follows that 
(5.39) IP(r)l _< Cr for 0 < r < 5. 
By (5.17) we further deduce that 
(5.40) IU(r)l _< Cr u for 0 _< r _< & 
As in the derivation of (5.38), these estimates allow us to prove that 
(5.41) IA(r,P(r),U(r)) - #o(A)l <_ Cr for 0 < r _< 5. 
We rewrite (5.34) in the form 
(5.42) 
r-°()p(r) _ s-O()p(s) 
s r #O(A) (rl_«(.x) _ 81_a(A)). 
= (f(p,P(p),U(p))- #o(A))p-«()dp+ 1 
If 1 < er(A) < 2, then f: If.x(p,P(p),U(p)) - #o(A)lp-«('X)dp < oc, and the desired 
assertion follows by an argument similar to that used in the case er(A) = 1. It 
remains to consider the case er(A) > 2. By (5.42) and (5.41) it follows that 
#o(A) 
(51--a(A) __ 
I(5-«)"(5) -r-«(X)P(r)) 1 

< CISU-o() _ 

for 0 < r < 5. Hence we have 
.o(A) ri _< Cr 2 
I P(r) 1---A) 
or equivalently (by (5.21)), 

for O<r<_5, 

for O<r<& 
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It follows, by Lemma 5.2 (1), that 
e3()2[ _< c 3 for 0 _<  _< 6, 
U(r) 
and by Lemma 5.2 (2), t, hat 
]f(r,P(r),U(r)) - po() - pl()r] < Cr  for 0 < r < 6. 

Repeating the above bootstrap argument step-by-step, we arrive at the estimate 

it follows that the linfit 
( Z +1() ) 
w=limr -() P(r) - 
-o (m + 1)! rr 
m=l 

exists and, by a similar argulnent as belote, (5.26) holds. 
The argument for the case 7,+1 () = 0 (i.e., a(X) = n) is similar. 
Finally, it is iather immediate to verify that if (P(r), U(r)) satisfies (5.17) and 
one of the equations (5.23)-(5.27), then it also satisfies (5.6)-(5.8). [] 

The first main result of this section is as follows: 

Theorem 5.3. Let A -# A. Then the followin9 hold: 
(1) If either fl(A) > 0 orTl(A) < 0 (== a(A) < 0), then the system (2.19)-(2.21) 
has a unique local solution satisfyin 9 conditions (5.1)-(5.3). 
(2) If ff(A) < 0 and "h(A) > 0 (¢= er(A) > 0), then the system (2.19)-(2.21) 
bas a 1-parameter family of local solutions satisfyin 9 conditions (5.1)-(5.3). More 
precisely, we have: 
(i) If"¢.(A) > O, 7+1(A) < 0 (¢=: n-1 < a(A) < n)forsome positive inte9ern, 
then for every real number w the above system has a unique local solution satisfying 
the following conditions: 
=  + wr 1+"() + 0@ "+1) as r - O, 
rrt=O 

Z.()r. 
tI'Itl (  ) r2+a( A ) 
(5.44) u(r) = , 
m=l  -[- 4 -[- E()) -[- O(r n+2) as r -- O; 
moreover, these are all the solutions of (2.19)-(2.21) satisfying (5.1)-(5.3). 
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(ii) If 3'n+l(A) = 0 (=: a(A) = n) for some positive integer n, then for every 
real number w the above system has a unique local solution satisfying the following 
conditions: 
(5.45) 
p(r) --  0-)--? "m'- ÷ I.tn_l(A)r n+l logr + wr + + O(r+21ogr) as r -- O, 
m=O 

(5.46) u(r)= E fl'(A)r'm! + 

#n- 1 (A)KM (A) rn+2 log r + &r n+2 + O(r n+3 log r) 
n+4 

as r -- O, 

where 

KM(A) (w #n-l(A)) 
n+4 n+4 
n 
1 
+ (n+4)(n+l),( -k(D+I)(A)+ E (n+lm)k(ï+l)(A)a'(A)) ; 
rn-=0 

moreover, these are ail the solutions of (2.19)-(2.21) satisfying (5.1)-(5.3). 

Pro@ By Lemma 5.1, we only need to prove that the systeln of equations formed 
by (5.17) coupled with one of the equations (5.23)-(5.27) bas a unique solution 
in the class CIO, 5], for some small 5 > 0. We shall use the coutraction mapping 
principle to prove this. 
The argulneut for the system of equations (5.17), (5.23) is simple, and we onfit 
it. Since (5.24) and (5.25) are special situations of (5.26) aud (5.27), respectively, 
it suffices to consider the systems (5.17), (5.26) and (5.17), (5.27). 
We first consider the system (5.17), (5.26). For small 5 > 0, we denote by ., 
the set of continuous functions (P(r), U(r)) defined on [0, 51 satisfying the following 
conditions: 

(5.47) P(r)- m! rr-I <- (1 + Iwl)r «() for 0 < r < 5, 
m=2 

/(A)r_  1+ for O_r_5. 
(5.48) U(r) - m! - 4 + 
m=3 

As in the proof of Lemma 5.2, one can show that if 5 is small enough, then 

(5.49) 

If(r,P(r),U(r))- _..(a)r I <_ 
m:O 

for 0 < r _< 5 and for all (P(r),U(r))  2. Here and later on we use C to denote 
various positive constants depending only on 5, A, w and n (but independent of the 
specific functions (P, U)). We now introduce a metric space (X, d) as follows: The 
set X consists of continuous functions (P(r), U(r)) defined on [0, 5], satisfying the 
inequalities 
(5.50) IP(r)-  °(A-----)-)r--wr «()1 < JIr , 
m! - 

m=2 
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Al2r TM , 
,=3 " 4 + a(A) , 
for 0 _< r <_ 5, where AI1 = C/(n - a(A)), C being the constant in (5.49), and 
is another constant to be specified later on. The distance function d is defined by 
IPI (r)- P2(r)l lUi(r) 
d((P1, U1), (P, U)) = 0<<sup rO(X ) + 0<<sup rt+O()_ o , 
where 0 < 0 < 1 is an arbitrarily chosen number. Clearly, (X, d) is a complete 
metric space, and X C ) if 5 is suflîciently small. 
Consider the mapping -" (P, U) --* (/5, U) defined by 

(5.52) 

(5.53) 

where (P, U) 6 ). We claim that (/5, U) 6 X. Indeed, using (5.49), it follows 
immediately that/5(r) satisfies (5.50). To see that U(r) satisfies (5.51) we set 
AA(r) =  KM(cA(p)) 
m + wP«() p3dp 
+  mx(O)oad 0 _  r_  wK() r+(  
. 4+«(a) " 
If we replace KM(cx(o)) and mx(O) (from (&10)) by their Taylor exposions of 
order up to  and reeall the definition of () from (5.20), we eily find that all 
the powers of order <  + 1 eaneel out. Henee, there exists constat Ç > 0 such 
that 
I()1  G rn+l 

for 0 < r _< 5. It follows that 
Or)- Z /rn(z)rrn--1 -- rl+a('k) 
=3 m! 4 + a() 
< V IK^,(c(o))l P(p)- Z am(A) pm-t 
m! -- wP«() p3dp + I%(r)l 
 rn.= 2 

__ ( C21I C1) rn+l 

for 0 < r _< 6, where C2 denotes the maximum of IK1(c(r))l for 0 _< r _< 6. 
Takin_g AI 2 to be the constant on the right-hand side of the la.st inequality, we see 
that U(r) satisfies (5.51). Thus (/5, U)  X and - maps ) into X. 
Next, we take any (P1, U1), (P2, U2) in X and set 

(P,U) = (P,U), i = 1,2. 
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Il is easy to verify that for any chosen 0 e (0, 1), 
sup If(r, Pl(r),Ul(r))- f(r,P(r),U(r))[/rl+«()-° 
<_ Cd((P1,U1),(P2, U2)). 
This enables us to deduce that 
sup IÏ(r) m Ï2(r)l/r °() < C52-°d((P,U),(P2, U2)). 
O<r<_5 
Il is also readily seen that 
sup loi(r)-U2(r)l/r+«()-° <_ CS°d((P,U),(P2, U2)), 
O<r<5 

d(([91,U1),(P2,U2)) <_ CS°d((P1,U1),(P2, U2)). 
Thus  is a contraction mapping in X, provided 5 is sufficiently small, and con- 
sequently,  has a unique fixed point in X. 
The proof in the case "n+l(A) = 0 (z er(A) = n) is similar. [] 
Remark 5.1. The analytic solutions constructed in §4 for A e (0, 1)\S coincide with 
the solutions (5.43), (5.44) corresponding to w = 0. 
From Theorem 5.3 we sec that in the case "h(A) _< 0, the solution (p, u) of 
(2.19)-(2.21) satisfying p, u e C[0,5] and p'(O) = 0 is Ulfique. If we weaken these 
regularity conditions, we can also gel a 1-parameter family of solutions, as in the 
case ?I(A) > 0 and f(A) < 0. More precisely, we have the following result: 
Theorem 5.4. (1) Suppose that /() = 0 (z a(A) = 0). Then for any w e R. 
the problem (2.19)-(2.21) bas a unique solution (p(r),u(r)) forO <_ r <_ 6, for some 
small 5 > O, satisfying 
(5.54) p, ?.tecl[0, S] and p'(0)=w. 
(2) Suppose that?l(A) < 0 (¢= -1 < er(A) < 0). Thenfor anyw e R. the 
problem (2.19)-(2.21) has a unique solution (p(r), u(r)) for 0 <_ r <_ 6, for some 
5 > O, satisfying the following conditions: 
(5.55) pe C[O,5]çC(O,5], u  C[O, 5]; 
p(r) = a(A) + wr +«() + O(r) as r - O, 
(5.56) u(r) =/3(A)r + wK^t(A)r2+«() + O(r ) as r -. O. 
4 + tf(A) 
Proof Consider the change of variables 
(5.57) p(r) =a(A) + rl+a(')P(r), u(r) : r((A) + U(r)), 
where P, U e CI0, 6] fq C  (0, 6] and they satisfy the initial conditions 
(5.58) P(0) =w, U(0) =0. 
One can readily verify that, under the transformation (5.57), the equations (2.19) 
and (2.21) are respectively changed into the equations 
(5.59) U'(r) + 3-U(r) = r«()KM(c(r))P(r) + mA(r), 
r 
(5.60) P'(r) = ](r,P(r),U(r)), 

so that 
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where mA(r) is as in (5.10), and ](r,P, U) is given by 
( g(0) uP 
](,..P,U) = - 
_ g;(r) r g;(O)p 
where 9;(r), h;(r) are  in (5.12) and (5.13), respectively. Since -1 < «(A)  0, 
the terre r«(;)KM(c;(r)) in (5.59) and ]; are integrable. Also, the terre UP/r is 
integrablc if U(r) converges to 0 at an appropriate rate as r  0. This enables us to 
transform the problem (5.58)-(5.60) into the following system of integral equations: 
(. () = . 
(5.62) P(r) = + ](p,P(o),U(o))do. 
Using the contraction mapping principle  in the proof of Theorem 5.3, we can 
prove that there exists  > 0 such that the system (5.61), (5.62) h a unique 
continuous solution (P(r), U(r)) for 0  r  , satisfying 
IP(r)l  M,, IU(r)l 
for some constants 3I and 312. Hence the desired result follows. 
Later in 8 we shall sec that the solutions of (2.19)-(2.21) constructed in this 
section are unique also within the cls of nonnegative weak solutions (see Threm 
8.). 

6. CONTINUOUS DEPENDENCE OF SOLUTIONS ON THE PARAMETERS 
By the results obtained in the previous section, for every 0 < A < A the 
problem (2.19)-(2.21) has a continuum of solutions depending on a real parameter 
oz. From (5.23)-(5.27) and (5.62) we see, at least heuristically, that, for a fixed 
the solution depends continuously on A for 0  A  A, A  S1, where 
S ---= {A G (0, A,)  -),,(A)---= 0 for some n _> 2} 
= {,k Ç (0,,k)  o-(,k)= m for some integer m >_ 1}. 
However, from the structure of the solutions (in Theorem 5.3) it is clear that as 
,k   for some  Ç SI, the solution does not have a limit. The nmin purpose of 
this section is to show that we tan reparameterize the solutions for 0 < ,k < ,k by 
a new parameter  in such a way that when oz = ¢z(,k, ) the solution will depend 
continuously on ,k for all 0 < ,k < , as well as on b. 
We introduce the set 
A = {  (0,)  "a(A) > 0} = {  (0,)  r() > 0}, 
which is clearly an open set containing S. By Lemnm 3.3, the following two 
situations nmy occur: (1) A is a single interval (,k0, ,k), where ,k 0 is either equal to 
zero (in the case er(A) > 0 for all 0 <  < ) or equal to the unique positive root 
of the equation o'(,k) = 0 (in the case that o'(,k) changes sign only once); (2) A is a 
union of finitely many disjoint open intervals, one of which has the form 
where h0 is the largest root of the equation er(A) = 0. For any   A and oz  R, we 
denote by (P(r; , w), U(r; A,¢z)) the solution of the system consisting of (5.17) and 
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one of the equations (5.23)-(5.27) (in accordance with the corresponding condition 
on A), and for any A E (0, A)\A we denote by (P(r; A, w), U(r; , w)) the solution 
of the system of equations (5.61) and (5.62). 
Consider Case (1) first. By Lemma 3.3, the set S1 consists of a finite number of 
points 1, 2, "", m, which are the roots of the equations V(A) = 0, V3(A) = 0, 
 -., 0_(A) = 0, and the sequence {A} 0 as described by Lemma 3.3. We 
rearrange the union of the two sets of these As as a monotone increing sequence 
and denote it as 
(6.1) , 2 "",, "'" 
By Lcmma 3.3 we know that lim  = A. For each integer n  1 we take a 
small number  > 0 such that all the intervals 
[1 -- 1,1 +1], [2 -- 2,2 + 2], "'" , [n -- n,n @ n],''" 
are contained in (Ao, Aoe) and do hot intersect each other. Let ,t = «(). Clearly, 
if (A) is monotone increasing, then m+l = m + 1; for genera.l (A) we only hve 
[mn+ -"mn[  1. We introduce, for every integer n  1, a function 
(A e [ - e,  + e], ¢ e R) as follows: 
(6.2) n('¢): ¢--(Smn_l()/(,n--6())) if 0<lA-le. 
We arbitrarily take a continuous function wn,n+ (A, ¢) defined for 
+ - en+ and ¢ G R and monotone increing in ¢ such that 
(6.3) 
n,n+l (n+l -- En+l, ¢) : n+l (n+l -- En+l, 
For Ao < A   - el we take 
(6.) o,1(,¢) :¢- (n,--l()/(l -- 6())). 
Finally, in case Ao > 0 we define 
(6.5) wo(,¢)=¢-o() (0< o, ¢R). 
Now define, for all 0 < A < A and ¢  R, a function w(A, ¢) by 
{ Wo(A,¢) if 0 < A  Ao, 
wo,1(,¢) if Ao<A1- 
(6.6) w(,¢)= .(,¢) if ,-e, <<,+e,, =1,2,..., 
w,+l(A,¢) if +eA+I-e+, n=l,2,--- 
It follows that w(A, ¢) is continuous for A e (0, A)S1 and ¢  R, and is monotone 
increasing in ¢. Notice however that w(A, ¢) is not continuous at   S1, since 
li w(A, ¢) = 
Consider Case (2) next. We write 
A = (o, o2) U (o3, o) ""  (o2-1, o)  (o, )- 
In the last interval we can define the function w(A, ¢) in a way similar to (6.6). 
Eve interval of the form (Ao2i-1, Ao2i) contains at most a finite number of points 
from S1. We write these points in increing order, say, 
-1, 2, "'" , - 
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As before, for every integer 1 _< j _< m we t.ake a small positive number ej such 
that ail the intervals 

are contained in (A02i-1, A02i) and do hOt intersect each other. We then define 
w(A, ¢) for A E [ij - ci, ïij -1- Ci] and ¢ E R (j = 1,2,..- , m) similarly as in (6.2), 
and, for every I _< j <_ m- 1, we arbitrarily define the values of w(A, ¢) for A  
(ïij + ai, ij+l - Q+I) and ¢ E R such that it is continuous on 
[o + ej, i+ - ej+l] x R 
and lnOnotone increasing in !b- For A E (A02i-1, il - e) U ( + e, A02i), we 
definc w(A, b) silnilarly as lu (6.4). Finally, in the COlnplementary set of A, which 
consists of the ration (0, Ami U lA02, A03] U [Ao4, A05] U..-U [A02k, Ao], we define 

for ¢eR. 

Clearly, the flmction w(A, b) (A  (0, A), b  R) defined in this way is continuous 
for ail A  (0, AOe)\S and b i R, but hot for A  SI, and is nlonotone increasing 
in ¢. 
Having inlroduced the function w(A, b) (0 < A < A,  e R), we now define 
(«7) ç¢() = ç(;A,(A.¢)), U¢() 

and 
(6.s) ¢() = .(A) + P¢(), .¢() = ((A) + u¢()) 
for aIl0<A<Aoe and¢R. 
The main result of this section is the following: 
Theorem 6.1. For any 0 <  < A and   R. there exists a corresponding  > 0 
such that for any (A, Ib) in a neighborhood of (A, lb), the solution (p¢(r), u¢(r)) 
exists for all r  [0, ($], and 

(6.9) lira_ _ p),¢(r) = pÇb(r), lira_ _ u),¢(r) = uÇb(r ) 
(,¢)-+(,¢) (,¢)(,) 

uniIoml Io  e [0, al. 

We shall only give the proof for the case (A, )  S x R; the proof for the case 
ï  (0, A)\SI (in part.icular, the case where a() = 0) is similar but simpler. We 
need the following lemma: 
Lemma 6.2 Let   S,,   R. Then there exist positive constants , C such 
that forO < lA- l << 1 and I¢ -l « 1, th« solution (P¢(r),U¢(r)) «xiÆts for 
all O N r  5 and 

(6.10) 

rn=2 

rn -- ra()) I Crn+l 
- #,_(A) -g o(A) < ' 

where n = cr(); the constants , C are in&pendent of A and ¢. 

Pro@ Notice t.hat since  = S, n = «() is an integer > 1. In t.he following, when 
we say "A near ", we mean that 0 < lA - 1 « 1, and similarly for 
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Since P-x¢(r) = P(r;A,w(A,¢)) and w(A,¢) = ¢- ttn_I(A)/(?I-OE(A)) for A near 
, we see that P-xg,(r) satisfies the equation 

(6.11) 

As in the proof of Theorem 5.3, we then have 

 rn -- ra(-x) , 
(6.12) P(r) = wr «(-x) + m! n 

as r -- 0, where n' = n fol" il -- 1 (O'(A) ( 7l, and n' = n + 1 for n < a(A) < n + 1 
Thus for A near  and for ail ¢ E R, file function 

(6.13) 

A-xb(r ) = sup IP-x.o(p)-  
O<p<_r m:2 

converges to zero as r --+ 0. Let 50 > 0 be a fixed small number, and let. 5-x¢ be the 
largest number in (0, 50] such that the solution of the systenl (6.11), (5.17) exists 
for 0 _< r < 5-x¢. We can clearly write, for 0 < p _< r < 5-x¢, 

(6.14) 

IP-xb(p)-  (tra(A) p mIl n--- 
., I <-   x-x¢(). 

By (5.17) we further have 

(6.15) 

Iu(»)-  ra(A)pra-1 < CDn+½a.x*(/)) %-CD n+l 

<_ Cp'+½ A¢(r) + Cp '¢+1 

for 0 < p _< r < 5-x¢; here and in what follows we use C to denote various positive 
constants independent of A and . We introduce the functions 

n 

so that 

(6.16) 

k Om(A) rm_ 1 
n÷l 
m! + 
m=l 

(recall that ]I(A) =/(A),/2(A) = 0). By (6.14), (6.15), 
(6.17) I,-x(p)l  pn l  a  } ( r ) , ]w-x¢(p)l  CO n+½ Z-xgar) %- Cp n+l 
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for 0 < p < r < 5. It follows (see the Appendix) that 

(6.18) 

n--1 
= pny,x¢(p) q- Zl,X¢(p)W,X(p) q- pz2,X(p)v,X¢(p) -- 

where 

(6.19) lY,W(P)I  C, IZl),¢(p)l  c, Iz2¢(p)l  c 
for 0 < p < ,¢ and for N,  respectively near  and @. Hence, 
 C2n-î¢(r) + Cn+x¢(r) + Cp n. 

By the second inequality in (6.18) it follows that 
[U,(p) + ()] >_ Ifi()l- 
-> I()l- ( C + c+ a¢()). 

Since limT--,0 A¢(r) = 0 and ff(A) # 0, we infer that for every (. b) near (, b), 
there exists a corresponding positive number ¢ _< ¢ such that for 0 < r < 

(6.21) 
Thus 
(6.22) 

Cr 2 ÷ Cr z¢(r)  -I()l- 

1 
IU,(,) + ,a()l >_ 5l.(.X)l >_ c > o 

for 0 < p < ¢ and for A,  respectively near  and t}. In what follows we use 
the saine notation  to denote the suprenmm of ail ¢ such that (6.21) holds for 
0 < r < ¢. Substituting (6.22) into (6.20), we get 

n-1 
m=O 
<_ Cp2n-1/kî¢(r) + Cpn+½ /k¢(r) + Cp n 

and then, by (6.11), 

for 0 < r < ¢. The terre with the coefficient p,n-l(A) can be estimated from the 
inequality 
I r'-r° <Cr½ 
Il--O" 
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(for r > O, a _> n- 1/4), and this leads to 
(6.24) 
<_ Cp'+ Aî(p) + Cp2A(p) + Cp + Cp¼ 
3 1 
<_ CPAî(r) + Cpz 
for 0 < p _< r < , where , is sufficiently near  such that [a(,) -ni < 1/4, and ¢ 
is near . By (6.13), it then follows that, for 0 < r <  and for ,,  respectively 
near  and , 
/(T) <_ cT /() + c¼. 

Thus, if 0 < r <  and 
(6.25) 
then 

4Cr  . Cr ¼ < 1, 

_ -- 1- 1-4Cr.Cr¼ <2Cr¼. 
2Cr 
We now replace the terre A(r) in (6.21) by the upper bound obtained in (6.26) 
and consider the inequality 
(6.27) Cr  + C . 2Cl 5 1()1. 
Since [(A)[ h a positive lower bound for  near  and the left-hand side converges 
to zero as r  0, we tan find 5  (0, 50) small enough so that (6.27) holds for all 
0 < r 5 5 and  near . By taking 5 sufficiently small we may sume that (6.25) 
aNo holds for 0 < r  5. We now use the maximality of  and 5 to prove that 
for ,  near  and , respectively. Indeed, we have either  = 5 or  < 5. 
Suppose first that  = 5. If 5 < 5, then, by (6.22), (6.23) and (6.26), 
Pe(r) and, consequently, Ue(r) are bounded for 0  r 5 5, and Ue(r) + 
() stKvs away from zero uniformly for 0  r  5. This allows us to extend 
(Pe(r), Ue(r)) beyond r = 5  the solution of a regular ODE system, which 
contradicts the maxinmlity of 5. Hence 5  5 holds in the ce where  = 
5. Suppose next that  < 5. Then by the maximality of , the equality 
in (6.21) must hold for r = . Since the second inequality in (6.26) holds also 
for r = min{5, }, it follows, by (6.27), that 5 < . Having thus completed the 
proof that 5  5, the estimate (6.10) follows from (6.23) and (6.26). 
Proof of Theorem 6.1. We shall only consider the case   S; the ce 

(0,))\S can be treated similarly. 
We first prove that for 5 as in Lemma 6.2, 
(6.28) lim_ _ P(r)= PMi(r), 
(,)(,) 
uniformly for all r  [0, 3]. 
From Lemma 6.2 we sec that 

3 
(6.29) IP(r), N Cr 

u() = u() 
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for 0 <_ r _<  and (A,,) near (A,¢), which fim.her implies, by (5.17), that 
7 
(6.30) IUxw(r)l < 
for 0 N r N  and (£, ) near (£, ). It follows that 
(6.31) fx(r, Px(r),Ux(r))  C 
for 0 N r N  and (, ) near (, ). Using these estimates ill (5.71, (5.81, we obtain 
(6.32) IU;(r)[  C, 
(6.aa) IPw()l  c- + c. 
By (6.32), (6.33) and (6.29), it follows that the N.milv of functions 
is equicontilmous and UlfifOl'lnly bomded for 0  r  3. We now sert that 
(6.a) 
(x,)(X,) 0<< (x,)(£) 0<< 
Indeed, otherwise there would exist a lmmber e > 0 and a sequence (,) 
(m = 1.2,-.. ), converging to (, ), such that 
(6.35/ lllE 
0<r<6 0<r<6 
for ail m. Since {P(r)} and {U(r)} are both equicontilmous and mil- 
fornfly bounded for 0  r  e, there exists a subsequence of {(,)}, which 
for simplicity is again denoted by {(,)}, such that P(r) and U(r) 
converge uniformly, in [0, 3], to some fimctions P(r) and U(r), respectively. Taking 
(A,) = (,) in (5.171, (6.11) and using the fact that 
lira (r n - r)/(n - ) = rn logr 
and the Lebesgue domilmted convergence theorem, we conclude,  m  oe that 
P(r), U(r) form a solution for the system of (5.17), (5.27). Since the solution of 
the system (5.17), (5.27) is unique, we get 
a contriction to (6.35). Hence (6.34) holds. By (6.34) and (6.8), (6.9) eily 
follows. 

7. GLOBAL PROPE1RTIES OF THE SOLUTIONS 

In this section we study the global properties of solutions of the free boundary 
problem (2.19)-(2.22) and the initial value problem (2.19)-(2.21). 
We first consider the global properties of solutions of the free boundary problem 
(2.191-(2.221. 

Theorem 7.1. Let (p, u, R,) be a solution of the free boundary problem (2.19)- 
(2.22) with p(r), u(r) in C'[0, Rx]. Then 
(1) p'(r) > 0 for 0 < r < R; 
(2) 0 < p(r) < 1 for 0 <_ r < Rx, p(Rx) = 1: 
(3) (u'(r) + ;u(r)) > o foO <  < n; 
(4) u(r) < 0 :for 0 < r < t,; 
(5) ,'(o) < o. ,.'(R) > o. 
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Pro@ First we note that the assumptions that p, u E Ci[0, RA] and u(0) = 0 
imply, by (2.19), that u 
We begin by proving that p'(r) >_ 0 for all 0 _< r _< RA. Assume that this 
assertion is false, namely, there exists an ro G (0, RA) such that p'(ro) < 0. Denote 
by (ri, r2) the lnaximal interval contailfing ro such that p'(r) < 0 for all r G (ri, r). 
Then either ri = 0 or 0 < ri < ro and p'(rl) = 0. In both situations we get 
u(rl)p'(rl) = 0. This implies, by (2.21) and Lemnm 3.1, that 
p(T1)---- O(CA(T1)). 
Similarly, we have either r -- RA, or r0 < r < RA and p(r) = O. In both 
situations we get u(r2)p'(r) = O, so that 
p() = 
Since p'(r) < 0 for r  (ri, r), it follows that. p(rl) > p(r2). On the other hand, 
since a(A) is monotone increasing in A (by Lêmma 3.1) and cA(r) is monotone 
increasing in r, we bave a(cA(rl)) < a(cA(r)), which is a contradiction. Hênce 
p'(r) >_ 0 for 0 < r < R» A silnilar argulnênt shows that p'(r) cannot he identically 
zero on any intervals, so that p(r) is strictly nonotone increasing. 
The assertion (2) follows inmlediately ri'oto the above result, because by Lelnma 
3.1 we have p(0) = a(A) > 0 and p(RA) = (1) = 1. Differêntiating the equation 
(2.19), we get 
 (,() + 2,()) = -/(cA())c() + ç(cA())c()p() +/^(cA())p'() 
>0 for O<r<_RA, 
because KD < O, c A > O, t;,^t > O, p > O, K^t > 0 and p' > 0, so that (3) holds. 
Next we prove assertion (4). Assume that this assertion is falsê. Since clearly 
u(r) cannot be identically zero, it follows that two situations may takê place: either 
(a) max0<<R u(r) > 0 or (b) u(r) <_ 0 for all 0 < r <_ RA, min0<<R u(r) < O, 
and there exists an ro G (0, RA) such that u(ro) = 0. Ill case (a) we let U(o) = 
maxo<r<R, u(r). It follows that u'(o) = 0 and U(o) > 0, so that 
2 
 '(o) + =(÷o) > 0. 
0 
On the other hand, denoting by (ri, r2) the nmximal interval containing o such 
that u(r)  0 for ail r E (ri, r2), we have (r2) -- 0 and u'(r2) _ O. It follows that 
'() + -() _< 0. 
2 
This contradicts assertion (3). Sinfilarly, in case (b), by comparing the values of 
'(r) -F (2/r)u(r) at the points fo, r, where r2 
mino<_< u(r), we again get a contradiction. Hence assertion (4) follows. 
We now prove that p'(r)  0 for 0 ( r ( RA. Indeed, sincep(r) _ 0 and 
u(r) < 0, we have u(r)p'(r) <_ O, so that, by equation (2.21), 
/vcA()) + (/^(cA()) -/(cA()))p() -/'^(cA())p() _< 0 
for 0 < r < R» It follows that 
(7.1) p(r) > 
This further implies that if p(ro) - a(cA(ro)) for some r0  (0, RA), then p'(ro) = 
a'(cA(ro))c(ro) > 0. It is now easy to show that for any r  (0, RA), p'(r) cannot 



3566 SHANGBIN CUI AND AVNER FRIEDMAN 

be zero. Indeed, if p'(r) = 0 at some point r in (0, R), then p(r) = a(c(r)), 
by equation (2.21), so that p'(r) > 0 by the preceding observation, which is a 
contradiction. Hence p'(r) > 0 for ail 0 < r < R. 
Finally, we prove assertion (5). Since u(r) < 0 for 0 < r < R and u(R) = O, 
we sec that u'(R) >_ O. Assume that u'(R) = 0. Then u'(R)+ (2/R)u(R) = 0, 
so that, by assertion (3), 
for all 0 < r < R. Multiplying both sides by r 2 and integrating over If, R] yields 
r2u(r) > 0 for all r E (0, R), which is a contradiction to (4). Hence u'(R) > O. 
Similarly we can prove that u'(0) < 0. [] 

Remark 7.1. (1) The conditions p E Cil0, RI, "t/ C1[0, RI imposed in Theorem 7.1 
can be replaced with the weaker conditions p  CI0, RI ç C I (0, R), u  C[O, R] C 
C 1 (0, R). Actually, these conditions can be weakened even further. See Corollary 
8.2 in the next section. 
(2) From the above proof we see that the inequality (7.1) is strict for 0 < r < R» 
If )  [), 1), then the solution of the initial value problem (2.19)-(2.21) sat- 
isfies u'(0) =/3(A) _> 0. and thus, by Theorem 7.1, it cannot be a solution of the 
free boundary value problem (2.19)-(2.22). Therefore, in the sequel we shall only 
consider solutions (pa, ua) for  E (0, ). 
We write for brevity 
p(r) = p(r), u(r) = u(r) (0 < r <_ 5). 
Since p(0) = a(A)  (0, 1), u(0) = 0 and u'(0) =/(A) < 0, we can extend (in a unique 
way) the solution to a maximal interval 0 < r < R such that 
0 < p(r) < 1, .(r) < 0 
for ail 0 < r < R, and either R = R, or R < R and one of the following three 
cases occurs: 
(i) u(R)=0; (ii) p(R)= 1; (iii) p(R) =0. 
We shall call (p, u) together with the maximal interval [0, R] a semi-entire solution 
of the initial value problem (2.19)-(2.21). The following result extends Theorem 
7.1 to semi-entire solutions of the initial value problem. 

Theoreln 7.2. Let   (0,)), and let (p, u, R) be a semi-entire solution of the 
problem (2.19)-(2.21). Then one of the following four situations occurs: 
(a) R <_R and 

u(r) < O for O < r < R, 
u(R) = 0; 

furthermore, p(R) = a(c(R)) < 1 if R < R, and p(R) = 1 if R = R. 
(b) R <_ R and 

((r)) < p(r) < , p'(r) > 0 for 0 < ç < t, 
u(r) < O for O < r <_ R, 
p() = 1. 
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(c) R < R and there exzsts fo  (0 R) such that 
c(c(r)) < p(r) < o(c(r0)) for 0 < r < fo, 
0 < p(r) < c(c,x(ro)) for ro < r < R, 
{ >0 for 0_<r<r0, 
p'(r) = 0 for r = fo, 
<0 for ro<r<_R, 
u(r) <0 for O < r <_ R; 
furthermore, either R < R, p(R) = 0 or I-¢ = R, 0 < p(R) < 1. 
(d) R < I-¢ and 
O < p(r) < oe(A), p'(r) < O for O < r < R, 
u(r) <0 for O < r <_ R; 
furthermore, either R < R, p(R) = 0 or R = R, 0 <_ p(R) < 1. 
Proof. If u(R) = 0, t.hen silnilar argmnents as in the proof of Theorem 7.1 show 
that Case (a) holds. It renmins to consider the case u(R) < 0. If p'(r) >_ 0 for 
0 < r < R, then Cse (b) follows as in Case (a). Suppose lmxt that p'(ro) < 0 for 
some r0 E (0, R). As before, we denote by (ri, re) the maximal interval in [0, RI 
containing r0 such that p'(r) < 0 fol" all ri < r < r2. V'e claire that r2 = /?. 
Indeed, if r2 < R, then p'(r2) = 0, so that by (2.21), 
(7.2) p(r2) = c(c,,,(r2)). 
Since we also bave u(rl)p'(r;) = O, no matter whether r > 0 or r = O, it follows 
that 
(7.3) p(rl) - Oz(CA(ri)). 
However, (7.2) and (7.3) contradict the fact that p'(r) < 0 for ri < r < re. Hence 
re = R. But,then if £1 = 0, then Case (d) holds, whereas if ri > 0, then, by a 
sinfilar argument as before, p'(r) > 0 for ail 0 < r < ri, and Case (c) holds. [] 
Corollary 7.3. In cases (a) and (b) of Theorem 7.2. we bave 
 (++ç/) >0 o0«<. 
dr 
Furthermore, m case (a) we also bave that u'(R) > O. 
The proof is immediate. 
Remark 7.2. From (2.19) we notice that, in case (a), 
u'(R) = -KD(c(R)) + IçM(c(R))c(c(R)), 
so that c(c(R)) > KD(c(R))/Kt(c(R)). This gives a lower estimate for R. 
A semi-entire solution (p, u, R) with R < Ra can be uniquely extended to a larger 
interval [0, R + 5] for some 5 > 0. To see this we consider the system of equations 
(2.19), (2.21) on the interval [R, R + 5], with initial values p(R) and u(R). Suppose 
first that (p,u, R) is as in case (a) of Theorem 7.2. Then p(R) = c(c0) (c0 -= ch(R)) 
u(R) = 0, and u'(R) > 0 (by Corollary 7.3). Furthennore, as will be shown in §8 
(see (8.10)), p'(R) exists and, in fact, 
(m) v'(n) = {;(c°) + (/,-,,(c0) - z,(c0))(c0) - z,,(c0)(c0)}c(n) 
n'(R) + 2IçM(Co)C(Co) - (h'M(Co) - KN(c0)) 
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Hcuce we can make the transformation 
;(r) = (c0) + (r - R)(p'(n) + P()), .(.r) = (- n)(u'(n) + u()) 
with P, U satisfying P(R)=0 and U(R)=0. We then get a system of differential 
equations which is, by the tact u'(R) > O, similar to that considered in Lemma 5.1 
(1} and Theorem 5.3 (1). By similar arguments as before we can then extend the 
solution uuiquely with u(r) > 0 to R < r _ R + 5. Notice that, by (7.4) and the 
last two equalities of (4.10), we clearly have p'(R) > 0, so that if 5 is small enough, 
then p'(r) > 0 for R < r _ R + 5, which further implies that (c0) < p(r) < 1 for 
R<r<R+5. 
Suppose next that (p, «, R) is as in one ofthe cases (b)-(d). Then, since u(R)  O, 
the system of equations (2.19) and (2.21) is a regular system at the point r --- R. 
We can therefore apply the classical theory of ODEs to uniquely extend the solution 
to a larger interval [0, R + 5], although the condition 0 _ p(r) _ 1 may no longer 
be satisfied for R < r < R + 5. 
Thc following theorclu asserts that the semi-entire solution can be further ex- 
tended to the entire interval 0 _ r _ R,, unless it blows up at some point 
Theorem 7.4. Let k  (0, k) and let (p, a, R) be a semi-entire solution of the 
system (2.19)-(2.21). Assume that R < R,. Then we bave the following conclu- 
sions: 
(1) If (p,u,R) is as in case (a) of Theorem 7.2. then (p(r),u(r)) can be extended 
to the entire interval [0, R,], satisfying 
(7.5) e(«,(R)) < p(r) < c(«,(r)), p'(r) > 0 and u(r) > 0 
for R < r <_ R» 
(2) If (p,u,R) is as in case (b), then (p(r),u(r)) can be extended ezther to the 
entire interval [0 R,], satisfying 
(7.6) p(r) > l, Æ'(r) > O and u(r) < O 
for R < r _ R,, or to a hall-open interval [0, ) (R < [ _ R,) such that (7.6) 
holds for R < r < , and 
(7.7) lirn_ p(r) -- +oo, lirn u(r) = O. 
(3) If (p,u, R) is as in cases (c) and (d), then (p(r),u(r)) can be extended to the 
entire interval [0. R,], satisfying 
(7.8) rein c_(«,(r)) _ p(r) < 0 and u(r) < 0 
for R < r _ R,, where c_ is the negative root of the equation (3.1), i.e., 
(7.9) 
Proof. Consider Case (1) first. Let (R, ro) be the largest open interval contained in 
(R, R) such tlmt the solution exists and p'(r) > 0 and u(r) > 0 for all r e (R, r0). 
We claire that r0 ---- R, and (p(r), u(r)) is well-defined at r -- R» Indeed, by (2.21) 
and (2.19), the conditions p'(r) > 0 and u(r) > 0 imply that 
(«,(R) < p(r) < («,(r)), ]u'(r) + çu(r)l _ const. < oo 
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for R < r < ro, so that p(ro) = lim-o-op(r) and u(ro) = limo_o u(r) exist. 
Furthermore, since p'(r) > 0, by (2.19) we sec that u'(r) + (2/r)u(r) is monotone 
increasing in (R, ro), which ilnplies that (r2u(r)) ' > r2(u'(R) + (2/R)u(R)) > 0 
and, therefore, u(ro) > 0. It follows that the systeln of equations (2.19), (2.21) is 
regular at fo, so that p'(ro) exists. We claire that p'(ro) > 0. Indeed, if p'(ro) = O, 
then p(ro) = (c,(ro)). But then, since p(r) < (cx(r)) for R < r < ro, we infer 
that p'(ro) >_ '(cx(ro))c(ro) > 0, which is a contradiction. Hence p'(ro) > 0. It 
follows that if ro < R,, then we can extend (p, u) to a larger interval on which the 
conditions p'(r) > 0 and u(r) > 0 still hold, which is contrary to the maximality of 
fo. Hence ro = R,, and Case (1) follows. 
Consider Case (2). Since p(R) = 1 and u(R) < 0, it is easy to sec (by (2.21)) 
that p'(R) > 0, so that (7.6) holds for r larger than bnt near R. Let (R, ro) be 
the largest open interval such that the solntion exists and pr(r) > 0, u(r) < 0 
for ail R < r < fo. Then either p(ro) --- limro-op(r) exists and is greater 
than 1, or lim_o_op(r ) = +oc. In the first case it follows from (2.19) that 
u(ro) = limo_o v(r) also exists. Since p'(r) > 0 for R < r < ro, we have 
p(ro) > p(R) = 1 >_ c(c(ro)), 
so that the right-|mnd side of (2.21) is -¢: 0; hence u(ro) # 0 and, therefore, u(ro) < 0 
and p'(ro) > 0. Bnt t.hen r0 = R,x, for otherwise we can extend (p, u) to a larger 
interval such that (7.6) still holds, which contradicts the maximality of (r, ro). 
Consider next the second case, where p(ro - 0) = +oe. We claire that 
(7.10) lira u(r) : O. 
 --0 
Indeed, since u(r) < 0 for r < fo, by (2.19) we have 
lzt(r) > --[k'D(«h(T)) + Ik'M(«,k(r))p(T) - +OE 
as r  r0, so that u(r) is monotone increasing for r near r0. It follows that u(r) 
has a limit as r approaches r0. We claire that te _= limr_,o u(r) is eqnal to zero. 
Indeed, snppose te < 0 and let 
" ») for 0<r<ro, 
y(r) 
for r = ro. 
Clearly, y (5 C(0, ro] V C(O, ro), and by (2.21), 
Koe(c(r))+(h'^t(c(r))-h'N(c(r)))p(r)-h'^t(c(r))p2(r) 
lim y'(r)=- lira 
-_.-o -_-o (ï)p(ï) 
/'M (c,(ïo)) 

Hence 

so that 

y() = h'^«(c(0)) (0 - ) + o(r0 - ) as   r0, 

--(1+o(1)) as rro. 
P()= h^l(C()) o- 

By (2.19), we then get 
'() _ 

t (1+o(1)) as rro, 
0 --  
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so that u(r) -- -oc as r -- fo, which is a contradiction. This proves (7.10). Thus 
assertion (2) holds with/ = r0. 
Finally, we consider case (3). Since p(R) = 0 and u(R) < 0, by (2.21) it follows 
that p'(R) < O, so that (_(r) < p(r) < O, p'(r) < 0 and u(r) < 0 for r larger than 
but near R. Let (R, r0) be the largest open interval such that the solution exists 
and 
p(,-) < 0, u(r) < 0 
for R < r < r0. From (2.21) we infer that 
{ < 0 if a_(c,(r)) < p(r) < 0, 
»'(r) > 0 if p(,-) < _(c(r)), 

for ail R < r < r0, and this implies that 
p() > ,ni, _(c()) >_ 
R < -r <-r o 
for R < r < r0, so that p(r) is bounded. 

rein ._(c(r)) 
R<r<Rx 

Using (2.19), we find that u(ro) =- 

limr_+o u(r) exists and is negative and u(r) has a negative upper bound in the 
interval R < r < r0. Helme, by (2.21), p'(r) is bounded for R < r < r0, which 
implies that also p(r0) = limr p(r) exists. We clailn that p(ro) < 0. Indeed, if 
p(ro) = 0, then on the one hand, since p(r) < 0 for R < r < fo, p'(ro) >_ 0; but on 
the other hand, it follows from (2.21) and the fact u(ro) < 0 that p'(ro) < 0, which 
is a contradiction. Recalling the maximality of (R, r0), we conclude that r0 = R,x, 
and (7.8) holds for all R < r _< R,x. [] 
Remark 7.3. Every entire solution is analytic at ail points except possibly at r = 0. 
Indeed, at a point where u does hOt vanish, the analyticity of p and u follows 
immediately from classical results, and where u vanishes, the analyticity follows as 
in the proof of Theorem 4.1, noting that if u(ro) = 0 and r0 > 0, then u'(ro) > 0 
so that in the equations analogous to (4.9) the coefficients of p()(r0) are uniforrnly 
positive. 
Remark 7.4. We can prove that if (p, u) is as in (7.7), then there exists a constant 
n > 0 such that 
p(r) = R,-L_ r (1 + o(1)), u(r)=-2tçM(c,(f))gV/--r(1 + o(1)) 
as r -- /. Since we shall hot use this result, we onfit the proof. 
Definition 7.1. The solutions obtained in Theorem 7.4 are called entire solutions. 
We distinguish four types of such solutions in accordance with Theorem 7.4: 
(i) If 0 < p(r) < 1, p'(r) > 0 for 0 < r < R,x, and there exists an R  (0. R,x] 
such that u(r) < 0 for 0 < r < R, u(R) = 0, and u(r) > 0 for R < r < R,x, then we 
call the solution of the initial value problem (2.19)-(2.21) a subsolution of the free 
boundary problem (2.19)-(2.22); in particular, if R < R» then we call it a strict 
subsolution. 
(ii) If 0 < p(r) < 1, p'(r) > 0 and u(r) < 0 for 0 < r < R,x and p(R,) = 1, 
u(R,x) < 0 ( p'(R),) = 0), then we call it a supersolution of the free boundary 
problem (2.19)-(2.22). 
(iii) If p'(r) > O, u(r) < 0 for 0 < r _< R,x, and there exists R  (0, R,x] such that 
0 < p(r) < 1 for 0 < r < R, p(R) = 1, and p(r) > 1 for R < r _< R,x, then we call 
it a bounded upper solution, whereas if there exists/  (0, R,x] such that (p, u) is 
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only defined for 0 < r </, p'(r) > O, u(r) < 0 for 0 < r </, and (7.7) holds, then 
we call it an unbounded upper solution. 
(iv) Il: the solution is as in Theorem 7.4 (3), or equivalently, p'(ro) < 0 for some 
ro E (0, R), then we call it a lower solution. 
Note that, by the definition, a supersolution is also a bounded upper solution, 
and a solution of the fl'ee boundary problcm (2.19)-(2.22) is a subsolution. 
In §§9, 10 we shall prove that, for each A E (0, A), there exists a unique  = 
@(A)  R such that (p¢, u¢) is an upper solution for ¢ >  and a lower solution 
for ¢ < 2, whereas (p, u) = (p,, u,) is either a subsolution or a supersolution. 
Moreover, if (PI, ul ) is a subsolution and (p, u2) is a supersohltion, then there 
exists a A* between ) and A2 such that (p., u.) is a solution of the free boundary 
problem (2.19)-(2.22). 

8. WEAK SOLUTIONS OF THE INITIAL VALUE PROBLEM 

In this section we refornmlate the problem (2.19)-(2.21) as a system of integral 
equations, and use it to introduce the concept of weak solutions. The integral 
equation formulation presented here will enable us to work with weak linfits of 
solutions. 

Theorem 8.1. Suppose that p(r), u(r)  L a [0, RI fq C 1 (0, /] and they satisfy equa- 
tions (2.19), (2.21) for 0 < r < R, where 0 < R < R, and R is as in (2.18). 
Then p(r), u(r) also satisfy the equations 
-lfor (8.1) u(r) = (\- IX'D(C(p)) + KM(c(p))p(p)/p2dp, 
1 (Kp(c(p)) + (KB(C(p))- 
for 0 < r  R. Conversely, if p(r), u(r) belog to L[0, RI ad satisfy the system 
of equations (8.1), (8.2) for almost ail 0 < r  R. and 
(8.3) p(r)  O for 0 5 r  R, 
then, aer modifying the values of u and p on a subset of measure zero, p  
C[O.R]CI(O,R], u  C[0, RI, and they satisfy (2.19)-(2.21) for all 0 < r  R; 
furthermore, 
(i) p(0) satisfies the equation 
(s.) he() + («.() - ())p(o) - h'()p(o) = o; 
(ii) if7EEx(A) > 0, then p(r) is continuously dff««tb« t  = 0, d p'(0) = 0; 
if 7EE(A) = O, then p(r) is continuously differentiable at r = O, but p'(O) can be any 
real number; if 7EE(A) < 0 ( -1 < «(A) < 0), then the limit 
 = lira p(r)r -«( 
exists, and if  = O, then p(r) is continuously differentiable at r = 0 and p'(O) = O. 

Proof. Suppose first that p(r), u(r) satisfy (2.19), (2.21). Mutiplying (2.19) with 
r 2 and integrating over [0, ri (for arbitrary 0 < r <_ R) immediately yields (8.1). 
Next, multiplying (2.21) with r 2, integrating over [0, ri, and using integration by 
parts-and equation (2.19), we obtain (8.2). 
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Now suppose that p,u E LOe[0, R] and they satisfy (8.1)-(8.3) for ahnost ail 
0 < r _< R. We modify the values of u(r) on a subset of measuÏe zero by replacing 
it with the right-hand side of (8.1). Then u(r) is contimmus for all 0 <_ r <_ R, and 
u(0) ---= 0. Similarly, after modifying the values of p(r) on a subset of measure zero 
of the open set 

(8.5) 

o : { E (o,)  .() # o}, 

we may assmne that p(r) satisfies (8.2) and is continuous at ail r  O. From (8.1), 
(8.2) we fm'ther conclude that u(r), p(r) are continuously differentiable in O. Let 

(8.6) 

s = {r e [0,n]  (r) = 0}. 

We claire that 

(8.7) meas(E) = O. 
Indeed, if meas(E) > O, then nleas(E*) = meas(E) > O, where * is the set of 
Lebesgue points of E. Since at each point of E we bave 
ffor(-KD(c(p))+KM(c(p))p(p))p2dp=O (by (8.1)), 
it follows that, for almost ail r  E*, 
-K»((O) + KM(())p(r) = 0, 
'(()) + (K,(()) - ,(()))p() = O. 
Eliminating p(r), we get 
ç(r) = 'M(«()),>(()) + (K(«())- K(())),'»(()) = o 
a.c. in E*. On the other hand, if we denote by c* the unique positive number such 
that Kg(c*) = KQ(C*), then by writing ç ---- KBKp + (Kg - IçQ)KD, we easily 
verify that 9'(r) > 0 if c(r) <_ c* and ç(r) > 0 if c(r) >_ c*, so that ç(r) has at 
most one zero, contradicting the fact that ç = 0 on a set of positive measure. This 
completes the proof of (8.7). Since meas() = 0, we may redefine p(r) for r  E bv 
(8.8) p() = (()) (r e 
this does not change the fact that both (8.1) and (8.2) are satisfied for ail 0 < r _< R. 
With the definition (8.8) at hand, we shall be able to prove that p(r) is continuous 
for r  E. 
Differentiating (8.1) and (8.2), one easily finds that u(r), p(r) satis .fy equations 
(2.19) and (2.21) in O. Clearly, O consists of a countable number of disjoint open 
intervals. Let (ri, r2) be any one of these open intervals (so that u(r) = O, and 
u(r2) = 0 if r2 < R). Arguing similarly as in the proof of Theorem 7.2, we infer 
that p'(r) changes sign at most once in (ri, r2). It follows that p(r) has limits as 
r --+ ri -k 0 and r -- r2 - 0. By (8.1), (8.2) we have 
; (»(())+ (-(())_ »(())){)) 
p{) = 
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for ri < r < r2. Using l'Hospital's rule, we deduce that 
(8.9) 
IÇp(CA(T1)) + (IÇM(CA(T1)) -- I4[N(CA(T1)))p(TI + O) -- I4[ltl(CA(T1))p2(T1 + O) = O. 
By Lmmna 3.1 alld (8.8), it follows that 
p(T 1 -3 t- 0) : O(CA(T1) ) = p(T1). 
As for p(r2 - 0), if either r2 < R (===> (r2) = 0) or r2 = R and u(R) = 0, then 
a similar argument as above shows that p(r2 - O) = p(r2) = c(c(r2)). If instead 
r2 = R and u(R) # 0, then clearly p(R - O) = p(R). Hence, p is right continuous 
at r and left contimtous at r2. This further implies that u'(rl + 0) and u'(r2 - 0) 
exist. Using arguments similar to those used in the proof of Theorem 7.1, we can 
show, if either r2 < R, or r2 = R and u(R) = 0, that p'(r) > 0 and u(r) < 0 for 
ri < r < r2, and u'(rl+O) < O, u'(r2-O) > 0. Fromtheabovediscussionit 
follows, in particular, that if r0 is an isolated point iii , then p is continuous at r0. 
Consider next the case where r0 is an accumulation point of E. Since u cannot be 
identically zero at any intervals, O is dense in [0, RI so that r0 is an accunmlation 
point of O. Let {r,}__l be a sequence of points in [0, RI converging to r0. By 
splitting it into two distinct subsequences if necessary, we may assume that {rn}n= 
is either monotone increasing or monotone decreasing. Consider the case where 
{rn}n=l is monotone increasing; the monotone decreasing case can be dealt with in 
a similar way. Bv further splitting the sequence into two subsequences if necessar.v, 
we nmy assume that { n}n=l satisfies one of the following two conditions: 
(a) r, E E, n = 1,2,.--; 
(b) r, E O, n = 1,2,-.-. 
hl Case (a) we have p(r,) = a(cA(r,)), so that 
liln p(r,) = linl a(c(r,)) = a(c(ro)) = p(ro). 
Tt ---+ O0 Tt ---+ CX2 
In Cse (b) two situations nmy occur: 
(bl) There is 5 > 0 such that (r0 - 5, r0) C O; 
(b2) r0 is the limit of ml increasing sequence of points in E. 
In the first situation we know by the preceding analysis that lim_+o_0 p(r) = p(ro), 
so that 
lin p(,,) = P('o)- 

In the second situation we denote by (rn,r2n) the open interval contained in O 
such that Vl, < v, < v2, and u(v,) = u(v2,) = 0, n = 1, 2,--.. Since p is 
lnonotone increasing in (rl,, 
O(CA(Tln)) =p(Tln -[-0) < p(Tn) < p(T2n --0)=O(CA(T2n)), Tt= 1,2,--- , 

so that 

lira p(r,) = lira a(cA(r,)) = lira a(c(r2,)) = a(c(ro)) = p(ro). 

Hence p ks continuous at r0. 
Having proved that p is continuous at any point of E, we conclude that p  
C[0, R] and, consequently, u  CI[0, R]. (The assertion that lim0u'(r) exists 
can be easily verified using (8.1) and (2.19).) Furthermore, since p is monotone 
increasing in every open interval (rl,r) of O such that either r2 < R or r = R 
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and u(R) = O, and since p is also monotone increasing in , we conclude that if  
contains more than one point, p is monotone increasing in the interval [0, r*], where 
r* =sup{r- rGE}; 
if r* < R, thon either p is monotone increasing in [r*, R] or it changes monotonicity 
once, say, at the point r** G (r*, R), and p is increasing in [0, r**] and decreing in 
[r**, RI. It follows that u'(r)+(2/r)u(r) is monotone increing for either 0 < r  R 
or 0 < r 5 r** (if r** < R). This implies, by similar arguments  in the proofs of 
Theorems 7.1 and 7.2, that either  = {0} or Z = {0, fo} for some 0 < r0  R. 
 next prove lhat if E = {0, r0} (0 < r0  R), then 
(8.10) 
,2ÇLoV,(r ) = {Iç(Co)+ (Iç,(Co) - tç(Co))a(«o) - h',(Co)a(«o) }c(ro) 
u'(ro) + 2Iç/(«o)a(Co) - (K/(Co) - IçN(Co)) 
 I(ro), 
where co = ca(fo). (Note that I(ro) is lhe derivative of p at fo formally computed 
by differentiating equation (2.21) at r = fo and selting u(ro)p"(ro) = 0.) To this 
end we difl>rentiate (2.21) for 0 < r < fo and divide it by u(r) to get 
 b() 
(8.11) p"(r) + p'(r) -- u(r)' 
where 
a(r) = u'(r) + 2Içl(«(r))p(r) - (tçl(c(r)) - L'N(C(r))), 
b( r) = { Içé(c  (r) ) + ('1 (c (r)) - ' (c (r) ) )V(r) - Kt (c (r))p(r) }ci (r). 
Let  be an arbitrary point in (0, fo) and set 
A() = f (p) ' 
ap, 0 < r < fo. 
si« () = '(o)( - o) + o( - o)   + o ad '(o) > o, (o) > o (by 
(3.2)), it follows that 
(8.12) A(r) = a(ro) 
«(gl-l.(l+o(a))  +-o. 
We multiply (8.11) with e a() and integrate over [, ri to get 
(8.13) p,(r)ea() _ p,()ea(?) + f b(p) ea(O)d p 
-   
By (8.12) we bave 
() ,() (1 + o(1))   +  - o, 

where 

a(ro) 2Içt(Co)((Co)--(Iç^t(Co))--IçN(Co))) 
u -- u'(ro) 1 = u'(ro) > 0 (by (3.2)). 

It follows that (b(r)/u(r))e A() is integrable on [,ro). Hence, by (8.13), 

g = lin p'(r)e A(r) 
"¢'  "¢'0 --0 
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exists, and 
r ° b(p) eA(O) dp. 
(8.14) n = P'(¢)eA(e) + u(p) 
We claire that n = 0. To see this we assume the converse: n  0. Since p'(r) >_ 0 
for 0 < r < r0, we have n > 0. From the definition of n d (8.12) it then follows 
that 
(8.15) lira p'(r) : +oe. 
rro--O 
Ve now write, for 0 < r < r0, 
p'(r) = tçp(«(r)) + (IçM(«(r) -- KN(«(r)))p(r) -- K(«(r))p2(r) 
and bv l'Hospital's rule, 
2i2 oP'( r ) 
= in, (' + ('' - ')- q,)c() - (e-«- (', - '))'() 

-«o-0 '(T) 
However, by (8.15), the left-hand side is +oe while the right-hand side is -, which 
is a contradiction. 
Having proved that n = 0, we infer from (8.13) and (8.14) that 
(8.16) p'(r) = - ffr ° b(PA(P)d'/eA(), 
and, invoking l'Hospital's fuie again, (8.10) eily follows. 
Similarly we can prove that lim+0p'(r) = I(ro). Hence p(r) is continuously 
differentiable at r0. 
Finally, assertion (i) follows immediately from (8.9) by taking r = 0, and asser- 
tion (ii) follows fron (8.13) and 
(0) () 
..... 
'(0) Z(A) 
by using previous arguments. This completes the proof of Theorem 8.1. 
Corollary 8.2. Let (p, u, R) be a solution of the free boundary problem (2.19)- 
(2.22) with p,u  C[O.R] C(O»R). Then assertions (1)-(5) of Theorem 7.1 
hold. 
Pmof. Indeed, in the proof of Theorem 7.1, the condition that p(r) is differentiable 
at r = 0 and r = R is used Olfly to ensure that at these points u(r)p'(r) = 0 so 
that 
(S.7) p(0) = (A) d p(R) = (). 
By Theorem 8.1 we oee that (8.17) holds also if p  C[0, R]  CI(0, RA), u 
C[0» R]  C(0, R) and (p, u) satisfy (2.19)-(2.22). Hence the desired sertion 
follows. 
Definition 8.1. A pair of functions p, u in L[0, RI satisfying (8.1), (8.2) a.e. in 
[0, RI is called a weak solution of (2.19)-(2.21). 
Theorem 8.1 serts that if (p, u) is a weak solution and p  0, then it is also 
a clsical solution, that is, p  C[O,R] C(O,R], u  C[0, RI, and (2.19)-(2.21) 
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hold. However, if we remove the condition p _> 0, then the result may be different, 
as briefly discussed in the next paragraph 
The condition (8.3) was used to prove that p(rl -O) = a(c(rl)) and similarly 
that p(rl + O) = c(c(rl)), so that p(r) is continuous at any boundary point r of 
O. If p(r) changes sign, then it may occur that 
p(rl-0) =c(c(rl)) but p(rl +0)=a_(c,x(r)) 
(or vice versa), where c_(A) is the negative root of (3.1) (see (7.9)). Thus a weak 
solution need not be a classical solution. 
The following result supplements Theorem 8.1 in case (p, u) is a weak solution 
and p may change sign. 
Theorem 8.3. Let (p, u) and (p, U) be respectively a weak solution and a classical 
solution of the problem (2.19)-(2.21) with the same A,  >_ O. Suppose that 
(8.18) 
for O < r < , for some a > O. Then (8.18) holds for all O < r < R. where R is 
equal to : 
(i) the positive zero of U if (ri, U) is a subsolution, 
(ii) the blow-up point of fi if(ff, U) is an unbounded upper solutzon, 
(iii) R if (ri, U) is a bounded upper solution, or 
(iv) the first positive zero of 
 if (, U) is a lower solution. 
Proof. Let (0, r0) (r0 > 0) be the largest open interval such that (8.18) holds for 
all r 6 (0, r0). We claire that r0 _> R. Indeed, if r0 < R, then, since (8.18) holds 
for all r 6 (0, r0) and since u(r) is a continuous function (by (8.1)) and U(r) < 0 for 
0 < r < R, it follows that u(r) < 0 for 0 < r < r0 + (for some  > 0; r0 + < R). 
Thus, by the proof of Theorem 8.1, p(r) is continuous and (p(r), u(r)) is a classical 
solution for 0 < r < r0 + e. By uniqueness of classical solutions we deduce that 
p(r) = 6(r) and u(r) = a(r) for 0 < r < r0 + , which contradicts the maximality 
of ro. [] 

9. EXISTENCE OF SUBSOLUTIONS 

In this section we prove that for any A less than, but near, A, there exists a 
unique  = (A) such that (pxs, ux) is a subsolution. In the next section we shall 
prove that for any A near 0, there exists a unique ¢] = (A) such that (p5, u5) is 
a supersolution. These facts wil] be used to prove the existence of a solution to the 
free boundary problem. 
The following comparison lemma will play a crucial role: 

Lemma 9.1. Let (pl,u) and (p2,u2) be two different solutions of (2.19)-(2.21) 
with the same A  (0, Aoe), defined on the same interval [0, R) (0 < R _< R). 
Suppose that 

(9.1) p(r) > p2(r) and ul(r) > u2(r) 

in some small interval 0 < r < . Suppose further that 

(9.2) u(r) < 0, u2(r) < 0 and p2(r) > 0 

forO < r < R. Then (9.1) holds for all O < r < R. 
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Pro@ Assume that the assertion is not true. Then there exists an r0 E (0, R) such 
that (9.1) holds for 0 < r < r0 and either (i) pi(r0) > p2(ro), u(ro) = u2(r0), 
or (ii) p(ro) = p(ro), u(ro) > u(ro). (Note that the case pl(ro) = p(ro), 
u(ro) = u(ro) cannot take place, by uniqueness of solutions.) In the first case 
we have u(ro) > u(ro), by (2.19). On the other hand, from ul(r) > u2(r) for 
! 
0 < r < r0 and u(ro) = u(r0), it follows that u(ro) < u(ro), which is a 
contradiction. In the second case we bave u(ro)p(ro) = u(ro)p(ro), by (2.21). 
Since lu(ro)l > lu(ro)l, we get 
p(0) u(0) 
-- >1, 
p(0) u(0) 
so that Æ](ro) > p(ro). On the other hand, from Æ(r) > Æ2(r) for 0 < r < r0 
and p(ro) = p(ro), it follows that p](ro) < p(0), which is again a contradiction. 
Hence the desired assertion follows. [] 
Lemma 9.2. For any   (0, An) the following hold: 
(1) If for some Ç  R, (p, u(p) is a subsolution, then for all ¢ > , (p¢, u¢) 
is an unbounded upper solution, and for all ¢ < Ç, (pa, u,¢) is a lower solution. 
(2) /f for some Ç  R, (pa, ua(p) zs an upper solution, then for all ¢ > Ç, 
(p,¢, u,¢) is also an upper solution; in particular, if for some Ç  R. (pa, uaç) is 
an unbounded upper solution, then for all ¢ > , (p,¢, ua¢) is also an ubounded 
upper solution. 
(3) /f for some Ç  R, (pa,ua) is a lower solution, then for all ¢ < Ç, 
(p, u,¢) is also a lower solution. 
Proof. Assertion (2) follows quickly from Lenmm 9.1. Indeed, from the expressions 
(5.43)-(5.46) and (5.56) near r = 0 and the fact that w = w(A,¢) is monotone 
increasing in ¢ it follows that, if ¢ > , then 
(9.3) pa¢(r) > paf(r), ua¢(r) > ua((r) 
for r near 0. If (pa(r), ua((r)) is an upper solution, then paç(r) > O, so that by 
Lemma 9.1, the above inequalities hold also for all r such that both (pa¢(r), u,¢(r)) 
and (pa(r), ua(p(r)) are well-defined. 
To prove assertion (1), consider first the case ¢ > . As before, (9.3) holds 
for ail r such that both (p,¢(r), ua¢(r)) and (pa,(r), ua(r)) are well-defined. It 
follows that (pa¢, u,) is either a subsolution or an upper solution. We claire that 
(p,¢, u,¢) cannot be a subsolution. Indeed, if (pa, ua¢) is a subsolution, then, by 
(9.3), the zero/,¢ of u,¢(r) and the zero/, of ua((r) must satisfy the inequality 
/a¢ </a. Since (p,, uab) is a subsolution, pa(r) > c(ca(r)) for 0 < r </,,, 
so that 
which contradicts (9.3). Hence (p,¢, u,) is an upper solution, and it is either 
bounded or unbounded. If it is bounded, then u,¢ (r) < 0 for ail 0 < r < R,, which 
is a contradiction because u,¢(r) > ua,(r) = 0 at r = R,. Hence (p,¢, u,) is 
an unbounded upper solution. Consider next the case ¢ < . By assertion (2), 
(p,¢, u,) cannot be an upper solution, and by the result we have just proved, 
(p,¢,u,¢) cannot be a subsolution. It follows that (p,,u,¢) nmst be a lower 
solution. 
Finally, assertion (3) follows immediately from (1) and (2). [] 
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Lemma 9.3. For any   (0, ), the set 
ÇI(/)-----{/) R: (p),u)) is an upper solutzon} 
is bounded from below. 
Proof. If the assertion is hOt truc, thon. by Lemma 9.2 (2), for any ¢  R. 
(p,p, u,p) is an upper solution. Using Lemma 9.1, we infer that, as ¢ decreases, 
both p,p(r) and u,p(r) decrease, whereas the interval of definition of 
increases (for unbounded upper solutions). Since p¢(r) (¢  R) are ail positive 
and, by (8.1), 
u(r) >_ C > --oo, C ildependent of 

the lilnits 

p = lim pa,p, u = liln 
exist and, furthermore, 
(9.4) 0_<p<p, u<u,p for ail ¢R. 
On the other hand, replacillg (p, u) in (8.1), (8.2) with (pa, ua) and letting 
-oc, we sec that (p,u) is a weak solution of (2.19)-(2.21), and, by Theorem 8.1, 
(p,u) is also a classical solution. It follows that there exists a ¢0  R such that 
P = Pa,po and u = u,po, which contradicts (9.4). Hence the desired assertion 
follows. [] 

Lemma 9.4. For any A  (0, A), the set 
2(A) = {¢  R: (p,ua,p) is a lower solution} 
is bounded from above. 
Proof. Ifthe assertion is false, then, by Lemnm 9.2 (3), (pa,p, ua) is a lower solution 
for any ¢  R. By Theorem 7.4 (3), the functions {p,p(r) : ¢  R} form a bounded 
subset of L[0, Ra]. We can therefore find an unbounded increasing sequence 
{¢,}=1 such that the corresponding sequence {pa,p,}= is .-weakly convergent 
in L[0, Rai. Let p denote the limit function. By (8.1), the corresponding sequence 
{ua,p,},%1 is uniformly convergent on [0, Rai. Let u deuote the limit function. 
Replacing (p,u) in (8.1), (8.2) with (p,p, u,p) and letting n -+ oo, one easily 
finds that (p, u) is a weak solution of (2.19)-(2.21). We claire that there exists a 
  R such that, for any ¢ > , 
d 
(9.5) -rp,p(r) > 0 for 0 < r < 5 
for some 5,p > 0. Indeed, if a(A) > 1, then by (5.43), (5.45) we have 
() = () + ½=()= + o( =) as  - 0. 
Since, by (4.10), 
1 
() = (- {/,()+ (,,()- ())- ,%()()}c5(() > 0, 
w sec that (9.5) actually holds for all   R. If a(A) = 1, then by (5.45) (taking 
n = 1) we sec that (9.5) also holds for all   R, because by (5.22), 
1 
' - ;»,() ()}c() < 0. 
,o()- 2z) {/c()+ (/(^,() /,())- -'  " 
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Filmlly, if-1 < er(A) < 1, then (9.5) follows from (5.43), (5.54) and (5.55), provided 
¢ is suflïciently large. 
Having proved (9.5), we can now apply Lemma 9.1 to deduce that, if 
then 
¢,(.) > (.), «() > ,() 
 long  px(r) > 0. Assuming for simplicity that ¢1 > , we deduce, in 
particular, that 
px(r)px,(r) for all 0<r< and nl, 
where  is the first positive zero of dpx/dr. It follows that 
p(r) k Px, (r) 
so that, by Theorein 8.1, (p, u) is a clsical solution on the int.erval [0, l. By 
uniqueness of the solutions, we infer that there exists 0  R so that (p,u) = 
(Pfo, Ufo)- Now take a positive integer N large enough  that ¢N > 0, which 
ensures that, for some  > O, 
Px, (r) > PX,o ('r) for 0 
But since 
p¢. (r)  p¢.(r) for 0 < r < 
where ' is the first positive zero of dpx¢/dr, we have, by taking the .-weak hinit 
of the px¢., 
PX¢o(r) k P¢(r) for 0r', 
which is a contradiction. This completes the proof of the lemma. 
Rema'rk 9.1. Since px¢(r) is uififormly bounded for absolutely large negative , one 
can find a sequence   -oe such that px¢ (r) .-weakly converges in L  [0, Rx] 
and u¢ (r) uniformly converges for 0  r  R» The limits p(r), u(r) form a 
weak solution of (2.19)-(2.21). By similar arguments  in the above proof, one 
can show that this solution calmot be a clsical solution such that p(0) = a(A). 
Thus p(0) = a_(A), where a_(A) is the negative root of (3.1) (see (7.9)). One can 
further deduce that p(r)  0 for ail 0  r  Rx. 
Lemma 9.5. There exists a   (0, A) such that for ail A  (, A) the set 
3(A) = {  R" (px¢,ux¢) is a bounded vpper solution} 
is empty. 
P'roof. For any A  (0, 1) we introduce the function 
(.) =  - K»(c(;)) + K(c(;))(c(;)) ;d;, 0 < .  , 
and set wx(0) = 0. It is clear that wx depends continuously on A in the topolo 
of CI0, Rx]. Clearly, 
3r(cx(r)), 
where 3(') is the function defined in (3.6), so that by Lemina 3.2, 
wx(r)>0 for O<rRx, 
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and 

G(o) = () < o, 
a7 (G() + T (T)) > o, 0 <  < 
for 0 < A < A. Hence there exists a  E (0, A) such that for A 
w(r)<0 for 0<r</, w(r)>O for /<r_<R, 
for some point/» We claire that, for A  (, A), the set t2a() is empty. Indeed, 
if this is hot the case, then there exists a bounded upper solution (p, u) for this 
so that p(r) > a(c(r)) for ail 0 < r _< R. By (8.1), we then have u(r) > w(r) for 
0 < r _< R. Therefore, u(r) > 0 for/ _< r _< R. Since u'(0) = ff(A) < 0, so that 
u(r) < 0 for r near 0, we see that (p, u) is a subsolution, which is a contradiction. [] 

Lemma 9.6. Given   (0, ), there exists an entire solution of the problem 
(2.19)-(2.21) which is a subsolution if and only if the set 3() is empty. 
Proof. If (2.19)-(2.21) has a subsolution, then by Lemma 9.2 (1), the set 3() is 
empty. Suppose conversely that ll3(A) is empty. By Lemmas 9.3 and 9.4, () 
and lz(A) are bounded, respectively, from below and from above, and by Lemma 
9.2 (1) there is at most one ¢ such that (p, u) is a subsolution. Consequently, 
(A) and z(A) are both nonempty and, by Lemma 9.2, 
inf (A) = sup2(A) ---- . 
We claire that (p, u)  (p}, u.}) is a subsolution. Indeed, using Theorem 6.1, we 
easily see that the set f2(A) is open, so that   f2(A). To prove that 
we assume the converse: b  (). Let [0, H) be the domain of definition of (p, 
so that 
lira p(r) = 
By the uniform boundedness of_{p.x  b  f2(A)}, we can find an increasing 
sequence {¢n}n°= converging to b, such that pA, converges .-weakly in L°[0, 
to some function/3 E L°[0, R.x], and u.x, converges uniformly to some function 
fi  C[0, R.x]. Replacing (p,u) in (8.1), (8.2) with (pA,,uA¢,) and letting n 
one easily finds that (/3, ) is a weak solution of (2.19)-(2.21). Since limn_oe 
we conclude, as in the proof of Lemtna 9.4, that (p(r), u(r)) = (/5(r), (r)) for r near 
0, so that, by Theorem 8.3, 
p(r) = (r), u(r) = (r) for 0 < r < R. 
It follows that p(r) is bounded for r ¢ [0, R), which is a contradiction. Therefore, 
 ¢ (A). 
Since we have proved that  does hot belong to Qx(A) nor to f2(A), it follows 
that (p, u) must be a subsolution. [] 

By Lemmas 9.5 and 9.6, we immediately get the following result: 
Lemma 9.7. There exists a   (0, Aoo) such that for any A  (, Ara), there is a 
unique number Ç = (A) su«h that (pag,, uM],) is a subsolution. 
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10. EXISTENCE OF SOLUTIONS OF THE FREE BOUNDARY PROBLEM 
In this section we prove the existence of a solution of the free boundary problem 
(2.19)-(2.22). If for some A E (0, A) there is a  E R such that (PA, uA) is a 
subsolution (resp. supersolution), then we say that A is a subsolution point (resp. 
supersolution point). Similarly, if for sonle A  (0, A) there exists ¢  R such 
that (pAçp,uA@, RA) is a solution of the free boundary problem (2.19)-(2.22), then 
we say that A is a solution point. A subsolution that is hot a solution of the free 
boundary problem will be called a strict subsolution, and the corresponding A will 
be called a strict subsolution point. We introduce the sets 
BI = {A  (0, A) " A is a strict subsolution point}, 
/) = {A  (0, A)  A is a subsolution point}, 
B2 = {A E (0, A) : A is a supersolution point}. 
By Lemma 9.7, the set/)1 is nonempty; in fact, the proof of Lemma 9.5 shows that 
B is nonempty. Later on we shall prove that B2 is nonempty. 
Lemma 10.1. A is a supersolution point if and only if it is hot a subsolution point; 
in other words, 
(10.1) /)1 u B = (0, A), 
Proof. Lenlma 9.2 (1) shows that if A is a subsolution point, then A camot be a 
supersolution point. To prove the converse, we note that if £ is hot a subsolution 
point, then the set ft3(A) is nonempty (by Lemma 9.6) and is bounded below (by 
Lemma 9.3). Let 
 = inf f3(A). 
We claire that (p, u) -- (PA, uA) is a supersolution. Indeed, using Lemma 9.1 and 
Theorem 8.1, one readily finds that p and u are the monotone decreasing limits of 
PA¢ and uA¢, respectively, as ¢ -  + 0, ¢  f3(A), so that they satisfy 
p(r) > a(cA(r)), p'(r) > O, u(r) <0 for O<r<RA, 
and p(RA) >_ a(cA(RA)) = 1, u(RA) < 0. If p(RA) > 1, then for sufficiently small 
5 > 0 we can find a corresponding « > 0 such that 
p(r)- (cA(r)) _> 
for all 5 _< r _< R» By Theorem 6.1 and the standard ODE theory, it follows that 
if  is sufficiently close to , then 
1 
for 5 < r < RA, so that if also  < , then   ft3(A), which is a contradiction. 
Hence p(RA) = 1, and (p, u) is a supersolution. [] 
In the sequel we need to consider limits of sequences of functions 
(n = 1, 2,--- ). Since these functions are hot defined on a common interval, it will 
be convenient to make a transformation of variables (r,p, u, 
r 
It is easy to verify that equations (2.19)-(2.21) are invariant under this change of 
variables, but the interval [0, RAI is changed into the unit interval [0, 1]. Clearly, all 
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results established in previous sections are still valid for the transformed probleln. 
In the sequel we shall always write, for brevity, the variables r, p, u and  as 
respectively r, p, u and ch. It should be noted that the values p(1), u(1) in the 
new variables are respectively equal to the values p(Rx) and u(Rx)/R in the old 
variables. 

Lemma 10.2. The sets B1, B2 are open. 
Pro@ To prove that B1 is open we assulue the converse, that is, there exists a 
point   B1 that is the lilnit of a sequence of poiuts {Arn}rn=l that are not strict 
subsolution points. By Lenmm 10.1, for each A, we have a corresponding (Pro, u) 
that is either a supersolution or a solution of the free boundary problem. In both 
cases we have, for each m, 
c(cx,(r)) _< pro(r) <_ 1, p'm(r) >_ O, um(r) _< 0 for 0 <_ r <_ 1. 
It follows that {p,}= has a subsequence that is .-weakly convergent in LotI0, 1]. 
For simplicity we assume that {p}= is such a subsequence, and denote by p 
the limit function. Then {u }= converges uniformly to a function u  C[O. 1]. 
Clearly, 
o(c£(r)) <_p(r) <_ 1, u(r) <_0 for 0_<r_< 1, 
and p(r) is monotone nondecreasing for 0 _< r _< 1. On the other hand, replacing 
A, p and v in (8.1), (8.2) with, respectively, A,, p, and u, and letting m 
we sec that (p, u) is a weak solution of (2.19)-(2.21) with respect to A = . Since 
p >_ 0, (p, u) is a classical solution, by Theoren 8.1, so that the above properties of 
(p, u) imply that 
a(c(r))<p(r)< 1, p'(r)>O, u(r)<O for 0<r<l, 
and p(1) = 1. Purthermore, we bave either u(1) = 0 or u(1) < 0. Clearly, in the 
first case (p, u) is a solution of thc free boundary problem, and in the second case 
(p, u) is a supersolution, so that in either case  # B, which is a contradiction. 
Hence the set B 1 is open. 
Next we prove that B2 is open. Suppose that  e B2 and let (i5,,5) = 
(p,}, u,}) be the corresponding supersolution. Then we bave 
£(r) > c(«£(r)), (r) > O, ù£(r) < O for 0<r< 1, 
and 15£(1 ) = 1, fi£(1) < 0. It follows by Theorem 6.1 and Lemma 9.1 that we can 
find a  >  sufficiently near  such that 
p,(r) > c(c£(r)), u,(r) < 0 for 0 < r _< 1 
(and consequently also p'x(r) > 0 for ail 0 < r N 1). This implies that for any 
5 > 0 sufiîciently small, there exists a corresponding e > 0 such that 
pï(r)-o(c£(r)) >e, u£{,(r)<_-s for 5_<r_<l. 
By Theorem 6.1, it follows that for (A,¢) sufiîciently close to 
p,x¢(r) - c(c,x(r)) >  for ( <_ r _< 1, 
_ e, u(r) < _ -e 
which implies that (Px, ux) is a bounded upper solution. Hence for A in a small 
neighborhood of ï the set 123(A) is nonempty and, therefore, by Lemma 9.6, these 
A's belong to B2, so that B2 is open. [] 
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When A = O, the probleln (2.19)-(2.21) (in the new variables) becomes 
(10.2) u'(r) + çu(r) = --h'D(0)(1 -- p(r)), 0 < r < 1, 
(lO.3) (o) =0, 
(10.4) u(r)p'(r)=--{(t(Q(O)--t(D(O))+tçD(O)p(r)}p(r), 0<r<l. 
The set of all solutions of (10.2)-(10.4) is characterized by the following lemnm. 
Lemma 10.3. For ever 9 a E R. (10.2)-(10.4) has a unique solution satisfyin9 
p(r) = (0) + cor 1+«(0) + o(rl+«(°)), 
WKD (0) r2+«(O ) 
(10.5) u(r) =/3(0)r + 4 + «(0) + °(r2+a(°)) 
as r -- O; furthermore, (i) if a = O, then 
p() = (o), 2(,.) = 
for ail 0 < r < 1; (il) if w > O. then (p,u) is an upper solution; (iii) if w < O. then 
(p, u) is a lower solution. 
Pro@ The fornmla (10.5) and the uifiqueness follow as in the proofs of Theorems 
5.3 and 5.4. By direct computation one finds that (po(r), uo(r)) = (c(0),/3(0)r) is 
a solution of (10.2)-(10.4), so that there exists w E R such that it coincides with 
the solution given by (10.5), and clearly (p, u) = (Po, u0) for w = 0. Next, noticing 
that 1 + er(0) > 0, we sec that p'(r) < 0 for r near 0 if w < 0, so that (p,u) is a 
lower solution if w < 0. 
Consider next the case w > 0. Since p'(r) = (l+cr(O))wr°(°)+o(r °(°)) (as r-- 0) 
and u(0) =/3(0) < 0, we can find a nulnber ( > 0 such that 
(10.6) p'(r) > O, u(r) < 0 
for 0 < r < (. Let. (0, R) be the largest open interval such that (10.6) holds 
for r G (0, R). Then either lilnrRp(r) = +ce or limrRp(r) = a for some 
a(0) < a < ce In the first case (p, u) is clearly an unbounded upper solution. In 
the second case the right-hand side of (10.2) is bounded, so that b = limr u(r) 
exists, and clearly b _< 0. If b < 0 and R < 1, then we can extend (p, u) to a larger 
interval such that u(r) < 0 in this interval. Also p'(r) > 0 in this intervak for 
otherwise, by (10.4), p(f) = c(0) at the first point  where p'(f) = 0, which is a 
contradiction since p(0) = c(A) and p(r) is strictly increasing in 0 _< r _< . Hence 
(10.6) still holds on this interval, which is contrary to the maximality of (0, R). 
Thus either b < 0 and R = 1, or b = 0. In t.he first case (p, u) is a bounded upper 
solution. If we prove that the second case cannot occur, then the proof that (p, u) 
is an upper solution (when w > 0) will be completed. Suppose b = 0. Then bi)' 
(8.1), (8.2) and l'Hospital's rule, 
a = lira p(r)= lira J KQ(O)p(P)p2dp 
-n -[¢ f) KD(O)(1-;(p))p2dp 
= lira KQ(O)p(r) = KQ(O)a 
n KD(O)(1--p(r)) KD(O)(I--a)' 
so that a = c(O), which is a contradiction. [] 
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Lemma 10.4. The set B2 is nonempty. More precisely, there exists a Â E (0, 1) 
such that (0, Â ) c B2. 
Pro@ If the assertion is not true, thon we can find a monotone decreasing sequence 
{Am}=l converging to 0 such that each Ara is a subsolution point. Let (pro, um) be 
the corresponding sequence of subsolutions. Since {p,},__l is uniformly bounded, 
we may assume that pro .-weakly converges to some function p  L[O, 1]. It 
follows that u, uniformly converges to some function u 6 CI0, 1]. By a similar 
argument as belote, we easily infer that (p, u) is a solution of the problem (10.2)- 
(10.4) with p(r) monotone increasing. Using Lemma 10.3, we conclude that (p, u) 
either is a bounded upper solution or is equal to (Po, u0). It follows that u(1) < 0. 
On the other hand, since (pro, um) are subsolutions, we have Um(1) OE 0, so that 
also u(1) OE 0, which is a contradiction. [] 
Proof of Theorem 2.1. The sets B1, B2 are both open (by Lemma 10.2}, nonempty 
(by Lenmlas 9.7 and 10.4), and they are disjoint (by Lemma 10.1). Hence 
(0, Aoe) ¢ BI U B2. 
Recalling (10.1), we conclude that /1\B2 - O, so that there exists at least one 
solution point. [] 

11. UNIQUENESS OF THE SOLUTION 
In this section we prove the uniqueness of the solution of the free boundary prob- 
lem (2.1)-(2.6). We assume that there exist two different solutions (cl,p, Ul, R1) 
and (c2, p2, u2, R2), and derive a contradiction. 
By Lemma 9.2, RI # Rz, and we may take R1 < R» Introducing the functions 
Ci(r) = ci(tRi), pi(r) = p(rR), (r) - ui(rRi), 0_< r_< 1. i-- 1.2, 
R 
we find that the (i,Pi, fi)'s satisfy the system of equations 
ï(r) + 2-«(r) = RîF(ei(r)), 0 < r < 1. 
r 

(11.1) 
(11.2) 
(11.3) 

(11.4) 
(11.5) 

fti(r)ffi(r)=KP(i(r))+(KM(i(r))--KN(ai(r)))i(r)--KM(i(r))Pî(r), 0 < r < 1, 
(11.6) i(1) =0. 
Since R < R2, from (11.1) and (11.2) we get, by comparison, that 
(11.7) i(r)>(r) for 0_<r<l 
and, by the maximum principle, that 
(11.8) (1) < (1). 
Clearly 
(11.9) il(l) = 2(1) = l, 
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so that, by (11.3), 
(11.10) 
and, by (7.4), 

ai(1 ) =a(1)=/(B(1) 

{ tç'p(1)+ ( h't(1)- tç'N(1) )fi(1) - h't(1)fî(1) }ê(1) 

a (1) +2/X'M (1)/5i(1) -- (/X'M (1) -- KN (1)) 
--/X-Q(1)(1) 
2KB(1)+Kp(1) = Aa(1), 

lfo 
(11.13) ai(r) =  { - h'D(ei(p)) + Kat(ei(p))/si(p)}p=dp, 0 < r <_ 1. 
Since the function --KD(c)+ KM(c)a is strictly monotone increasing in both c and 
(, using (11.7) and (11.12) we infer that 
(11.14) 
--/X'D(I(p)) + KM(«I (p) )/51(p) > --tX'D(ê-2(p)) + KM(ê2(P))(P) for fo < r < 1. 
Thus, if ro = 0, then, by (11.13), 1(1) > u(1), which is a contradiction to (11.6). 
It follows that fo > 0, and then 
(11.15) /5(ro) =/5(ro), 
which implies, by (11.12), that 

(11.16) 

Since 

(11.13) gives 
ai(0) = --- 
so that, by (11.14), 

o 

al(r0) (2(/'0). 

Combining this with (11.16) and recalling that i < 0, f{ > 0 (by Theorem 7.1), 
we conclude that 
On the other hand, since the function 
t(p(C)+ (KItI(C)-KN(C) )Oe- KM(C)Oe2 = Kp(c)(1-oe)+ KM(C)Oe(1-oe)- KQ(C)Oe 

--! --! 
p(ro) OE p2(ro). 

1 { __ KD(i(P) ) q- t(hI(i(P) )i(P) } p2 dp = ai(l) = O, 

Let (ro, 1) be the largest open interval such that 
(11.12) /51(r) > i6(r) for ro < r < 1; 
By (11.3) we have 

O<ro<l. 

where A > 0. Recalling (11.8), we conclude that 
(11.11) i0(1) </5(1). 
From (11.9) and (11.11) it follows that there exists a 6 > 0 such that 
/51(r) >f2(r) for 1-6<r< 1. 
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is strictly monotone increasing in c (for fixed 0 < a < 1), we infer from (11.5), 
(11.7) and (11.15) that 
-(ro)p(.,-o) > 2(r)p(), 
which is a contradiction. [] 

Remark 11.1. The proof of the existence part of Theorem 2.1 shows that if A1 is a 
supersolution point and A2 is a subsolution point, then there exists a solution point 
of the free boundary problem in the interval with endpoints A, A2. Furthermore, 
points A ne 0 are supersolution points, and points 
are subsolution points. This fact, combined with the uniqueness part of Theorem 
2.1, implies the following: If A. is the ullique solution point of the free boundary 
problem, then every A  (0, A.) is a supersolution point, and every A  (A., A) is 
a subsolution point 
Remark 11.2. The assumption KD(1) = KQ(1) = 0 used throughout the paper tan 
be weakened as follows: 
L'D(1)  0, KQ(1) k 0 and KQ(1) < 
In this case, a(1)  1, so that for a solution point A, u(Rx) = 0 and 
a(1)  1, rather than p(Rx) = 1  in Theorem 2.1. Other than this and other 
similar differences, all the results of 4-11 hold with minor changes; for example, 
in Theorem 7.2 (a) one replaces p(R) = 1 by p(R)  1, and in 11 one replaces 
(11.10) by 
Ul(1 ) = u 2(1) = Ke(1)(1). 

(12.1) 
Then 

12. APPENDIX: THE PROOFS OF (5.30) AND (6.18)-(6.19) 
Lemma 12.1. Let  E (0, A) and a() >_ n, where n is an integer >_ 2. Let A be 
a number in a small neighborhood of . Assume that 
n+l 
P(r)--  O-4)rm--1-}- v(r), U(r) : E  rm-1 -t- (r). 
m=2 

(12.2) 

n--1 
rn=0 
= <() + zi(r),() + z()() - :M(c(r)),(), 

where 
(12.3) Iv(r)l _< C, Izl(r)l _< C, 
and C is a constant independent of v, w and A. 

Iz(r)l  c, 

Pro@ Let p(r)= ()+ rP(r). Since Oq()= 0, Pl()-----P(,,) and ()=0, we 
have 
() =  m()< 
 + v()  p() + (), 
m=O 
n+l 
() + u() =  
m + w(r). 
m=l 
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By (5.11), (5.12), (5.13) and (5.8) we see that 
(/3(A) ÷ U(r))fx(r,P(r),U(r)) 
= --/,'M(c())P() -- (/)()P() + («()/) 
+ (g(o)/())(1/)(() + u())P() 
= (V){>(«()) + (,(«())- -(«))()- 'M(«())()} 
- (() + )(/)(() + u())P() 
 I -I. 

Clearly, 
Il  

+ (/){ (KM(«()) - 
- 2KM(«())p())v() - KM(«()). rv()} 
 I + I. 
Using TayloFs expansions of the filnctions ff(c(r)), tçM(c(r)) and K(c(r)) 
u to order  about r = 0, and (g.t9), {0.21), and recalling (a.1) and the relations 
()  =K(a)ck(O)=O, i=D,M,N,P, 
 (al= .K(«())]=o 
we compute that 
: , -+( ,  () 
m=2 m:2 
2 
m 
+ ry(r), 
or, by expanding the products, 
n-- 1 
m (m+ ()() _ )())m+-() 
m=0 j=2 
m+2 m--j+2 
j=2 k=0 
j=2 
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so that, by the relation 
(12.4) 
(which follows from (3.8)) and the definition of #, (see (5.21) and (5.19)), 
n--1 n--1 mm m+l 
I, = Z(A)  p(A)r TM +  (m + 2),  (})Z(A)_+(A). 
m=O m=l j=2 
(2.5) 
 (e()- ,()) + () 
+ («()+ )()   
 J, + J + J + J, 
where yi(r) (i = 1,2,3) are functions depending only on a(A) (0  m  n), Kp, 
IçM, KN and c, so that they are uniformly bounded for ail A near . Later on 
we shall also use y(r), z(r), z2(r) to dcnote various functions possessing similar 

properties. Since 

we have 

= J2 ÷ rny4(r), 

Hence 

r 
rn=O 

n--1 
J : (U(r)- w(r)){ E Pm(A)rm + (c(A)+ 1)--l(P(r)r -- 
rn=O 

-- ]n_l()rn-l(u(r)- w(r)) -- r"y4(r). 
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where 

Recalling (12.4) and noticing that 

r Ir=o(KM(cA(r)) -- KN(cA(r)) -- 2KM(cA(r))pn(r)) = O, 

we easily find that 
112 - (a(A) + 1)/3(A). lv(r) = rz2(r)v(r) - Ix'M(c(r))v2(r). 
r 
Hence 

and the lemma follows. 

Formulas (6.18) and (6.19) follow inmediately from Lemma 11.1. To prove 
(5.30), recall that a(A) =/2() = 0, so that, by (5.28) and (5.29), (12.1) holds 
with v(r) and w(r) satisfying 
Iv()l < c , I(T)I-< Crn+' for 0 < r < 5. 
Hence the right-hand side of (12.2) is bounded by Cr . Since also 
I(A) + u()l _> I(A)I > 0 for 0 < r < 6, 
we see that (5.30) follows. [] 
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SHARP FOURIER TYPE AND COTYPE 
WITH RESPECT TO COMPACT SEMISIMPLE LIE GROUPS 

JOSÉ GARCA-CUERVA, JOSÉ MANUEL MARCO, AND JAVIER PARCET 

ABSTRACT. Sharp Fourier type and cotype of Lebesgue spaces and Schatten 
classes with respect to an arbitrary compact semisimple Lie group are inves- 
tigated. In the process, a local variant of the Hausdorff-Young inequality on 
such groups is given. 

[NTRODUCT[ON 
Let 1 _< p _< 2. An operator Sl)aCe E is said to have Fou,ier type p with respect to 
the compact group G if the vector-vahled Fourier transfornl exteilds to a co,npletely 
bounded map 
c,E ' L}(C,) -- E(C,), 
where p' = p/(p - 1) is tlle expoilent conjugate to p. That is, a vector-valued 
Hausdorff-Young inequality of exponent p is satisfied. Sinlilarly, if we replace the 
operator -C.E by its inverse, we get tlle notion of Fourier cotype p' of E with 
respect to G. Followillg the notation of [8], we define the constants 
C(E,G) II.,E Il «b(L (),£ (5)) and C,(E,G) --1 
(G))" 
The Fourier type and cotype become stronger conditions on the pnir (E, G) as 
the exponents p and p' approach 2. This gives rise to the notions of sharp Fourier 
type and cotype exponents. The present paper grew out of the project to investigate 
the sharp Fourier type and cotype of Lebesgue spaces L p and Schatten classes S p, 
and it is a natural continuation of [8]. However, as we shall see bclow, some other 
results have appeared in the process which are interesting in their own right. 
In Section 1 we recall that the natural candidates for tlle sharp Fourier type 
and cotype of L p and S p (where now 1 _< p < c) are min(p,p ) and max(p,p ) 
respectively. To show that this guess is right, one wouk| have to show that for 
l_<p<q<2, 
() C(L,(),C,) = C,(LÆ'(),C,) = , 
(b) C(LP'(),G) = Cq,(LP(),G) = oc, 
with the obvious modifications for the Schatten classes. In Section 2 we make some 
rernarks about (a) and (b). First we show that, to bave any chance of getting 
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positive answers to these questions, we have to require that the group G' not be 
finite and the operator spaces L p and S p be infinite-dimensional. Then, under such 
assumptions, one can easily get the following inequality: 
Cq(LP(fi),G) >_ limsup Cq(lP(n),G) 
and the analog for L p' (f). Therefore, the growth of Cq (IP(n), G) and Cq(lP'(n). G) 
provides a possible way to obtain (a) and (b). In the last part of Section 2 we 
analyze the vector-valued Lebesgue spaces and Schatten classes. 
The growth of Cq  (IP(n). G) is investigated in Section 3. To be precise, if G stands 
for a compact semisimplc Lie group and 1 _< p < q _< 2, then there exists a constant 
0 < IÇ(G,q) _< 1 such that Cq(lP(n),G) > IÇ(G,q)r 1/Æ-1/q for ail n _> 1. If one is 
able to show that /Ç(G, q) > 0, this result gives (a). Moreover, we would obtain 
optimal growth, since Cq(IV(n), G) <_ n I/p-/q for any compact group. We shall sec 

that 
IC(G,q) -- inf sup { Ilfll«a 

f central, f 

where {/A',  n >_ 1} is a basis of neighborhoods of 1, the identity element of G. The 
Hausdorff-Young inequality on colnpact groups yields/C(G, q) _< 1. The interesting 
point lies in the inequality tC(G, q) > 0, vhich constitutes a local variant of the 
Hausdorff-Young inequality on G with exponent q. 
Sections 4 and 5 are completely devoted to the proof of this local inequality. In 
the abelian setting, the particular case G = 2" was explored by Andersson in [1]. 
The basic idea is to consider a function f  2" -* C as a complex-valued function 
on IR supported in [-1/2, 1/2). Then, by expressing the norm of fon Lq'(IR) as a 

Riemann sure, one obtains 

where k(t) = k/qf(kt). This gives IC(E, q) > 13q, where 13q = v/q/q/q'/q' stands 
for the Babenko-Beckner constant, see [2} and [3], but in fact the equality holds, as 
was proved by Sj61in in [15]. g,b show here that Andersson's argument, suitably 
modified, is also valid in the context of compact semisimple Lie groups. In Section 
4 we summarize the main results of the structure and representation theory of 
compact semisimple Lie groups that will be used in the process. Then we use these 
algebraic results to get an expression for the Fourier transform of central functions 
f : G -* C in terres of the Fourier transform -T on the maximal torus T of G. 
This will allow us to work over the maximal torus where we know that Andersson 
obtained a satisfactory result. However, in the non-commutative setting, the degree 
d, of an irreducible representation 7r does not have to be 1. We shall sec that this 
becomes a further obstacle, to be treated in Section 5. There we combine some 
results, such as the Weyl dimension fornmla, concerning the representation theory 
of compact semisimple Lie groups with classical harmonic analysis to avoid this 
ditticulty. 
On the other hand, if we notice that Cq(l p' (n), G) = Cq,(lP(n), G), we can under- 
stand the growth of this constant as the dual problem of the growth of Cq(lP(n), G) 
in the sense that we replace the Fourier transform operator .Ta,w() bv its inverse. 
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Therefore, since the dual object is no longer a group (as it is when G is abelian) we 
do not have a Fourier inversion theorem and we should hot expect to reconstruct 
the proof given in Sections 3, 4 and 5 step by step. At the time of this writing, we 
are not able to solve this problem, and so we pose it as follows: 

Problem. Let G be any COlnpact senfisilnple Lie group and let 1 _< p < q _< 2. 
Does the estilnate C (l p' (n), G) >_ K(G, q)l/p-1/q hold for solne positive constant 
/C(G, q) depending only on G and q? 

Finally, we point to a non-connllutative notion of Radeinacher type for operator 
spaces, see [9]. We think this notion could be helpflfi iii order to study the growth 
of C(l p' (n),G). 

1. STATEMENT OF THE PROBLEM 
All throughout this paper sonle basic notions of operator space theory and llOll- 
commutative vector-vahled discret(, L p spaces will be assunled. The definitions and 
results about operat.or spaces that we are using can t)e found iii the book of Effros 
and Ruan [5], while tbr the study of our non-cOlnlnutative L  spaces the reader is 
referred to [12], where Pisier analyzes them iii dctail. In any case, all the analvtic 
preliminaries of this paper are Slillllllarized iii [8], where we studv the Fourier type 
and cotype of an operator space with respect to a conlpact group. In order to state 
the problenl we want to solve, we begin by recalling the definitions and the main 
properties of Fourier type and cotype. 
Let G be a compt topological group endowed with its Haar nleasure , nor- 
malized m that (G) = 1, and let n  Ç be gll irreducible unitary representation 
of G of degree d. Here the sylnbol Ç Stallds for the dual object of G. Given an 
operator spoee E, it was shown in [8] that, by fixing a bis on the representation 
space of each n  G, the Fourier transforln operator a,E for fllnctions defined on 
G and with vahles on E has the form f  LË(G)  (f(n))  E(Ç), where 
f(n) = f f(g)n(g)*d,(g) and E(Ç) =  hld. e E. 
Here kI denotes the space of n x  complex matrices. Let 1  p < oe. If S(E) 
stands for the vector-vahled Schatten cls on kI  E, we defilm the spaces 
 p X 1/p 
We write £P(Ç) for the case E = C Finally, let 1  p  2. Then by the Hausdorff- 
Young inequality on compoet groups (sec [8] or Kullze's paper [10]) it is hot difficult 
to hk tht 2,(L;(C) e «) c C;'(â) e « nd 2&ç(C;(â)e «) C L;'(C) e « 
This lnotites the following definitions. 
Definition 1.1. Let 1  p  2 and let p' denote its conjugate exponent. The 
operator spoee E bas Fourier type p with respect to the compact group G if the 
Fourier t.ransform operator 
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can be extended to a completely bounded opera.tor from LOe(G) into £e(). In 
that case, C(E, G) will stand for its «b notre. 
Definition 1.2. In the smne fashion, the operator space E has Fourier cotgpe p' 
with respect to the compact group G if the inverse 
G,E 
can be extended to a completely bounded operator from £(G) to LE(G ). As 
2 G). 
before, we shall denote its cb norm by Cp, (E, 
One of the properties proved in [8] is that every operator space has burier type 
1 and Fourier cotype ec with respect to any compact group. In particular, the 
COlnplex interpolation method for operator spaces (sec Pisier's work [11]) provides 
the following result. 
Lemma 1.3. Let 1 <_ Pi <_ P2 < 2 and assume th.at E has Fourier type P2 with 
respect to G. Then E has Fourier type p with respect to G. Similarly. Fourier 
cotype p' of E with respect to G irnplies Fourier cotype P'I of E with respect to G. 
Therefore, the Fourier type and cotype becolne stronger conditions on the pair 
(E, G) as the exponent p (and consequently its conjugate p') tends to 2. So Lemma 
1.3 gives fise to the following definition. 
Definition 1.4. The sharp Fourier type and cotype ezponents of an operator space 
E with respect to the compact group G are defined respectively by 
pl(E,G) = sup{p_<2"E hasFouriertype p withrespectto G}, 
p2(E,G) = inf{p' > 2  E has Fourier cotype p' with respect to G}. 
If E has Fourier type p (E, G) with respect to G, we say that E has sharp Fouiner 
type pi(E, G). The sharp Fourier cotype of E is defined analogously. 
In order to simplify the statement of the problem we shall need the following 
lemma (sec [8]) which analyzes the Fourier type and cotype of the dual E* of an 
operator space E with respect to a compact group G. 
Lemma 1.5. Let 1 < p < 2 and let p' be the conjugate ezponent of p. Then we 
have the equalitiesC(E*,G)= C,(E,  G) andC,(E*,G)=C(E,G). 
The problem we want to investigate in this paper is how to find the sharp Fourier 
type and cotype of Lebesgue spaces and Schatten classes. Concerning these top- 
ics, we present here a result given in [8] from which we start out. In what follows 
(f, ,A, ) will denote a a-finite or regular measure space and Sï the classical Schat- 
ten class over the space of compact operators on l 2. 
Theorem 1.6. If 1 < p < 0% then the spaces LV(f). S and S bave Fourier 
type min(p,p') and Fourier cotype max(p,p'). In fact. the vector-valued Fourier 
transform, or its inverse, is a complete contraction in each of the cases considered. 
Therefore, if we consider two exponents p and q such that 1 _< p < q _< 2, we 
would like to find conditions on G and 12 under which 
(a) C(LP(f),G) = C2q,(LP'(),G) = 
(b) Cq(LP'(f),G) = Cq,(LP(f),G) = 
with the obvious modifications for the Schatten classes. 
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2. SOME REMARKS ABOUT THE PROBLEM 

In this section we shall ,nake some remarks about the problem we have just 
stated. We begin by showing some necessary conditions that should hold to obtain 
a positive answer to our question. Second, we wonder about sufiïcient conditions 
that we shall work with in the rest of this paper. Finally, we study what happens 
if we consider vector-valued L p spaces (or Bochner-Lebesgue spaces) and vector- 
valued Schatten classes. 

2.1. Necessary conditions. The first necessary condition that we shall present is 
on the compact group G. We bave to exclude finite groups from our treatment, 
since, as we shall sec immediately, every operator space E has sharp Fourier type 
and cotype 2 with respect to any finite group. Anyway, the next result is a bit more 
accurate. 

Proposition 2.1. For a finite group G, every operator space E satzsfies the esti- 
ltatesCp(«,G), Cp,(«,G) _ ICI 1/p' fo 1 _ p _ 2. 

Pro@ Let us assume that C (E, G) <_ IGI 1/2 for every operator space E; then we 
have C(E,G) = C(E*, G) <_ ]GI */2 by duality. The desired estimates are then 
obtained by conq,lex interpolation from the equalities clé(E, G) = COe(E, G) = 1 
(proved in [8]) and the case p = 2. Therefore we lotus out attention on the case 
p = 2. It suffices to check that for all m > 1 and any fanfily of fimctions {fij " G -* 

Therefore we obtain 

and, since  d = [G[ by the Peter-çVeyl theorem, we are doue.  
Next we show that we cannot work with measure spaces (, , ) that are a 
union of finitely many -atoms. Before that we need to define the cb distance 
between two operator spaces. It is due to Pisier, and it constitutes the anMog of 
the Banach-Mazur distance between two Banach spaces in the context of operator 
space theory. Given two operator spaces E1 and E, we define their ch distance by 
the relation dcb(E1,E2) = inf{]]U]lcb(E,E=)]lU--]]cb(E=,E)} where the infimum runs 
over all complete isomorphisms u : E + Eu. The following result (also extracted 
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from [8]) relates the Fourier type and cotype of two operator spaces E and E2 to 
their ch distance. 

Lemma 2.2. Let 1 <_ p < 2 and let E, E2 be operator spaces. Then 
cp(E2, a)  dcb(El,F-,2)cp(E1, a) 

and 

C],(E2, G) <_ dcb(E,,E2)C,(E,G). 
Proposition 2.3. Let 1 <_ p < q <_ 2 and assume that (ft, .,4, oe) is a union of finitely 
many oe-atoms. Then every compact group G satsfies the following estimates: 
C(L,() C,) = C_,L''(),C,) < 
C(LP' (),) = C,(L(),) _< ()/«-/'. 
In fact, since 1/p - 1/q = 1/q  - 1/p , we have the saine bound for both cb nos. 

Pro@ Applying Lemma 2.2 and the last part of Theoreln 1.6, we get the estimates 
C(LP(), G) _< dcb(LP(l),Lq(l)) and C(LP'(t),G)q, _< dcb(L p' (Q),L q' ([)). On 
the other hand, it is straightforward to check that, for 1 _< Pi < P2 _< oc and such 
a measure space (,A, oe), we bave dcb(L m (Q), LP(ft)) <_ oe(çt) 1/p-Up. [] 

In other words, we do hot allow finite-dimensional Lebesgue spaces. Since 
the ch distance between two Schatten classes of the saine finite dimension is also 
finite, the arguments used in Proposition 2.3, Theorem 1.6, and Lernma 2.2 are also 
valid fo show that the only Schatten classes that could make Theorem 1.6 sharp 
are those of infinite dimension. 

2.2. Suflàcient conditions. The Fourier type and cotype of the subspaces of a 
given operator space E are bounded above by the respective type and cotvpe of E. 
The proof of this result is straightforward, see [8]. 
Lemma 2.4. Let 1 <t9<_ 2 and let F be a closed subspace of E. Then we have 
the estimatesC¢(F,G) <C¢(E,G) andC2p,(F,G)_C,(E,< G). 
After the conditions above, we shall work in the sequel with infinite compact 
groups and infinite-dimensional Lebesgue spaces and Schatten classes. Since the 
measure space (ff, .A, u) is no longer a union of finitely many u-atoms, we obtain 
that the n-dimensional space lP(n) is a closed subspace of LP(ft) for ail n > 1 and 
any 1 < p < oe. Moreover, recalling that the subspace of diagonal matrices of S is 
completely isomorphic to lP(n), we deduce that the saine happens for the Schatten 
classes S. Hence sharpness of Theorem 1.6 will be guaranteed if, for 1 _< p < q < 2, 
we bave 
(a') c((),) = c,(#(.,0.) --  as , - , 
(»9 C(# (n), a) = C,(t(n), a) --  as . - . 
Therefore out aire from now on will be the study of the growth of the constants 
Cq(lP(n), G) and Cq(lP'(n), G). The first remark about these constants that we can 
already make is that both have a common upper bound: 
(/P'(.), G) < -/P-/q 
This is an obvious consequence of Proposition 2.3. 
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2.3. Vector-valued spaces. Theorem 1.6 was also studied in [8] for vector-valued 
spaces. Here is the statement of the result obtained there. 
Theorem 2.5. Let 1 <_ p <_ oc and let E be an operator space having Fourier 
type min(p,p') (respectively, Fourier cotype max(p, p') ) with respect to G. Then the 
spaces L(), S(E) and S(E) bave Forier tp« i(pp') (ptV, Fo 
cotype m(p,p')) with respect to G. 
Let 1  p  oe and min(p,p ) < q  2. Let E be in Theorem 2.5. Then 
Lemma 2.4 gives the following estimates: 
C(L(),C) > C(L(),C), C(L(),C) > C(E,C), 
with the obvious modifications for the Schatten classes. Hence we have shown that 
sharp Fourier type or cotype of LP(fi) (respectively S) provides sharp Fourier 
type or cotype of LÉ(ff) (respectively S(E)) with respect t.o G. Also, the saine 
conclusion is obtained assuming sharp Fourier type or cotype of E. Iu particular, 
the sucient condition given above also works for vector-vahmd spaces. Therefore 
we focus our attention on the growth of the constants C(IP(n), G) and C(l p' (n), G). 

3. ON THE GROWTH OF Cq(lP(n),G). 
We shall assume in what follows that G is a compact semisimple Lie group. 
Semisimplicity is an essential assumption in the arguments that we shall be using. 
Anyway, for the moment, the only property of such groups that we shall apply 
is the existence of a maximal torus T in G. The following result gives, in par- 
ticular, statement (a) in Section 1, with the obvious modifications for Schatten 
clames, whenever we work with infinitdimensional operator spaces and compact 
semisimple Lie groups. 
Theorem 3.1. Let 1  p < q  2 and let G be a compact semisimple Lie group. 
Then there exists a constant 0 < (G, q)  1. depending on G and q, such that for 
all n OE 1, 
(G, q)l/p-1/q  Ç(lp(n), G)  l/p-1/q. 
In particular, we observe that the growth of C (IP(n), G) is optimal for compact 
semisimple Lie groups. The proof of this result starts by applying the existence of a 
mimal torus T to consider a countable family {gk " k  1} of pairwise commuting 
elements of G; just take gk G T. For every n  1 we take U to be a neighborhood 
of 1 (the identity element of G) satising 
gj gk =0 for l j,kn and j¢k. 
We recall here that we can always consider a central function f supported in  
and belonging to Lq(G). For example, take n to be invariant under conjugations 
(see Lemma (5.24) of [6]) and f = lu,where lu stands for the characteristic 
function of . Henceforth f will be a central function in Lq(G) supported in 
, to be fixed later. Then we define the function  - G  C  by n(g) = 
(f(gg), f(g2g),..., f(gg)). We obviously have the estimate 
c(t(), «)  
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Soit suffices to prove that this quotient is bounded below by IC(G, q)rt 1/p-1/q. The 
following lemma will be very helpflfl for that purpose. 
Lemma 3.2. Let 1 <_ p, Pz <_ oc, 7r E  and  > 1. Consider the matmz-valued 
v««tor A.,, = (rc(g),rc(g2) .... ,rc(g)). Thon 

COlllnlOll eigenvectors of r(g), 7r(g2) ..... 7r(g). 
lnatrices are diagonal: 

Pro@ Since 91,g2,-.-,gn are pairwise comnntting, there exists a basis of C d' of 
Therefore, in that basis, all these 

(k) = . 
Moreover, [01 = 1 for 1 _< 3 <- d, because ofthe mfitarity of (9)- Hence, applving 
the complete isometry (see Corollary (1.3) of [12]) between loe  (d) and the subspace 
of diagonal nmtrices of S  
 (E), we easily obtain the desired equality. 
A 
(i) The value of [I,H¢¢.)(O)-  begin by recalling that, since f is central, 
1 L 
by Schur's lenmm. Here  is the irreducible character sociated to w and 
1 denotes the idcntity matrix of order m x m. On the other hand. f(9') 
is the translation by ça of f; therefore, 
ê.(w) =  f.(9)k.(9)d.(9) (w(91),w(92),...,w(9)) 
So we get. by Lemma 3.2, the following equality: 

(il) The value of IIIIG(,()- w h,,e 

G n \ q/p \ l/q 
n q 1/q 
= (llS(k)llca)) = 
k=l 

since the sets {g/. 1/J n - 1 _< k _< n} are pairwise disjoint. 
In summary, we have obtained that C(IP(). G) >_ IC(G, q, n)n/P-1/q where the 
constant K(G, q, n) is given by 
IIïll,,,«c,) 
Iç( G, q,  ) = 
IIfll() 
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If we define IC(G,q) = infn_>l (G,q,), it is obvious that IC(G,q) _< 1 by the 
Hansdorff-Young inequality on compact groups. Thus it remains to check that 
/C(G, q) > 0. For that aire, since we have hot fixed fn )'et, we need to see that 

inf sup  f central, f E L q(G), supp(f) C L/, > 0. 

We shall prove this fact in Section 5 where we study the suI)remum of the Haus- 
dorff-Young quotient for central flmctions sui)ported in arbitrary small sets. As we 
shall sec immediately, semisimplicity of G will |)e essential in our proof. 

4. A SIMPLE EXPRESSION FOR THE FOURIER TRANSFORM 
OF CENTRAL FUNCTIONS 
In this section we apply solne basic results concerning the structure and represen- 
tation theory of compact semisimple Lie groups to provide a simple expression for 
the Fourier transform of central flmctions defined on snch groups. These algebraic 
preliminaries Call 1)e foun(I in Simon's bo(,k [14] or alternativel.v in [7], but we 811111- 
marize the main tol)ics here. Let. G he a compact semisimple Lie group and let 9 be 
its Lie algebra. In what follows we choose once and for ail an explicit maximal torus 
T in G, while b will stand for its Lie algebra. That is, b is the Cartan subalgebra of 
9. The rank of G will be denoted by r: in particular, T _ T  where T = IR/Z with 
its lmtural group structure. Also, as is custolnary, we consider the complexification 
9c = 9 ® i9 (with complex conjugates taken so that 9h¢ = {Z E 9,c " Z = Z} = i 9 
and similarly hh¢ = ih) with the complex ilmer l)roduct ( , ) induced by the Killing 
form. ,Ve also recall that the Weyl group I/Va associated to G tan be seen as a set 
of r x r unitary matrices IV (isometries on hn¢) with integer entries and det IV = +1. 
In particular, the set Wâ = {II :t  Il  )/VG} hecomes a set of isometries on h. 
The symbol T4 will stand for the set of roots, and. if we take Ho  1 such that 
a(H0) ¢- 0 for any root (. the symbol T4 + = {(  T4- a(H0) > 0} denotes the set 
of positive roots. Finally, we shall write Aw and ADW for the weight lattice and 
the set of donfinant weights, respectively. 
Now that we have fixed SOlne notation, let us consider a central function f  G --, 
Ç and a dominant weight A  ADW. By the dominant weight theorem there exists 
a unique 7r, E ( associated to A and, since f is central, we tan write, by Schur's 
lenlllla 
1£ 
f(r) =  f(9)X(9)d(9)le 
where d, is the degree of 7r,, )t, is the character of 7r, and 1 denotes the m x m 
identity matrix. We now recall the definition of the flmctions Ao appearing in the 
Weyl character formula. Given L+  Ij, we define the functions exp  Ij -- Ç and 
A+ " Ije --+ Ç by the relations 
exp/3 (H) = 
A+(H) =  det " exp (W(H)). 
The maxilnal torus T is isomorphic via the exponential lnapping to the quotient 
space he/Lw, where Lu- is the set of those H  he satisb'ing exp(2rriH) = 1 
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That is, Lw is the dual lattice of Aw. Therefore, the functions expz and AZ are 
well-defined functions on T if and only if/3 E Aw. As we know, the integral form 
1 
is not necessarily a weight, and so the functions exp and A6 could be not well- 
defined on T. To avoid this diflîculty we assmne for the moment that Gis simply 
connected: this condition on G assures that 5 G Aw. Hence, applying consecu- 
tively the Weyl integration formula and the Weyl character formula, we get 

= 1 /T f(t)A(t)A+(t)dm(t)ld 
dlcl 
where m denotes the Haar measure on T normalized so that re(T) = 1. Now, if we 
write A,x+ as a linear combination of exponentiMs, we obtain 
tI'EWG 
=  [ 
f(t)A(t) exp_(+)(t)dm(t) ld , 
d 
since A(W(t)) = det WA(t) and f(W(t)) = f(t). We recall that, taking coordi- 
nates with respect to the bis {w, w2,..., w} of fundamental weights, any ight 
A  Aw h integer coordinates. Therefore, we can understand the lt expression 
 the Fourier transform of fA on the maximal torus T evaluated at A + 5. Hence 
we have 
(1) [() = T(fA)(A 
+ 
5) ld, 
for f : G  C central and G any compt semisimple simply connected Lie group. 
When G is hot simply connected, a more careful approach is needed. We have 
W t (5)  5  Aw for M1 W  Wa- In particular, we note that 
expA+ =  detWexPw,(+)  
is a well-defined function on T for M1 A  Aw. This remark allows us to write 
xA  = (exp A+)(exp_A)  a well-defined function on T. Henceforth, 
applying again Schur's lemma, the Weyl integration formula and the Weyl charter 
formula, we get 
" E 
- d l(t)(exp_A)(t)exp_(t)dm(t)le 
where the lt equaliV follows from the change of variable t  Wt(t). That is we 
have shown that 
a 
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where Ba = exp_a A. This expression is now valid for any compact semisimple 
Lie group, and it coincides with (1) for simply connected ones. 

5. A LOCAL VARIANT OF THE HAUSDORFF-YOUNG INEQUALITY 
ON COMPACT SEMISIMPLE LIE GROUPS 

As we mentioned in the introduction, this section is devoted to the proof of a 
local variant of the Hausdorff-Young inequality on compact semisimple Lie groups. 
We recall that this result provides the relation (G, q) > 0 for 1 _< q _< 2, which we 
needed in Section 3. 

Theorem 5.1. Let 1 < q < 2 and let G be a compact semisimple Lie group. Then 
there exists a constant 0 < K;(G, q) _< 1 such that, for any open set lg C G, we bave 

: f central, f ¢ LU(G), supp(f) clg} >_ lç.(G,q). 

Since the norms of ]" and f on Cq'(G) and Lq(G) respectively do hot change 
under translations of f, we can assume, without loss of generality, that L/ is a 
neighborhood of 1. Before proving Theorem 5.1 we need some auxiliary results. 
Let us assume that G is simply connected and let f " G -- C be a central fimction. 
A quick look at relation (1) allows us to write 

(3) 

1 
]'(rx) = -- det H" .T(fA)(Wt(A + 5))ld 

for all W ¢ Wa. On the other hand, let us denote by P the hyperplane of 
orthogonal to a with respect to the complex inner product given by the Killing 
form. The infinitesimal Cartan-Stiefel diagram is then given by the expression 

Lemma 5.2. Let G be a compact semisimple simply connected Lie group. Then 
w« bave {Wt(A+5) " W e V«, A G ADw} = Aw\P. Moreov«r, th« mappzng 
(W, A)  W × ADw  Vt(A + 5)  Aw \ P is injective. 

Proof. Since G is simply connected, we have that {A + 5  A ¢ ADW} : Aw  C int. 
Here C stands for the fundamental Weyl chamber and C int for its interior. Now, 
since P and Aw are invariant under the action of M?â and for any Weyl chmnber 
C there exists a unique W  Wa such that W(C) = C, we obtain the desired 
equality. Finally, the injectivity follows from the uniqueness mentioned above. [] 

Proposition 5.3. Let G be a compact semisimple simply connected Lie group and 
let f  G -- C be a central function. Then there exists a constant .A(G, q), dependmg 

on G and q. such that 

EAw\P 

cET+ 
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Pro@ Since fis central and Gis simply connected, we can apply (3) to obtain 

AEADw 
$I$ E I/VG )EADw 
Morcover, the Weyl dimension formula for dA gives 

I/V E I/VG )EADW 

+ 5))flllllz,] 1/¢  

IT(f Az)(II't( A ÷ 6) )lq' ] l/q ' 
H I(, +)1 «-2 " 

E ÷ 

Finally, we observe that 
H I(,+)1 = H I(I(),+)11/2= H l( 
since any Il" E "VG is a pernmtation of the set of roots. Therefore, by Lemma 5.2, 

19($A)(A)I¢ 1/q' 

we have 
II'll:«(â) = [  
 E"R,÷ AAw\P 

The proof is completed just by taking 
A(a. q) = [ 1 

We are now ready to give the proof of Theorem 5.1 for simply connected groups. 
Let {H1, H2,..., H,.} be the predual basis of the fundamental weights; any element 
of Lw can be written as a linear combination of H, H2 .... , H with integer co- 
efiïcients. Then, since T oe tl/Lw, we can regard T as the subset of b given 

k=l 
On the other hand, let us fix a boulded central function fo : G -- C then fo 
can be understood as a function on T invariant under the action of 14)a. Now, 
since the Weyl group is generated by a set of reflections in [?, fo can be regarded 
as a complex-valued function on [?, supported in TE and symmetric under such 
reflections. Let us recall that {w, w2 .... , w } stands for the basis of fundamental 
weights. Let - : 1 - 2/q; the way we have interpreted the function fo allows us 
to define the function 
L-(foA,) " O ---, C 

by 

I,-(foA,)()= H I( ",«)1" 9c»(f°As)() where = «:E«w«" 
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Remark 5.4. The motivation for the notation is that in a classical group such as 
SU(2) the function just defined is nothing but thc Fourier transform of the fractional 
integral operator 
IT(f)(x) - F-T) f(y)(x- y)T-ldy 
acting on foA. Hcre lies the lnain difference with the comnmtative case (where 
a Hausdorff-Young inequality of local type has been already investigated, see [1]) 
since the presence of the degrees d,x (as a product iii Proposition 5.3 by the Weyl 
dinmnsion formula) requires the presence of a factor of .TOR(foA ). This fact does 
not happen in the commutative case, since d,x --= 1 for all   ADw. 

Lemma 5.5. Let Gbe a compact semisimple simply connected Lie group and let 
f  G--* C be a central function. Then 9r0R(fA)(ç) ---= 0 for all ç  P. 

Pro@ If   P, there exists a root er such that  G P. Let. S« be the reflection 
in Pe; then .b(fA)() = detS« .b(fA)(S«(()) = -.b(fA)((), since, as we 
know, S«  Wâ. [] 

The flmction .b(foA6) is analytic, since foA6 has compact support and, by 
Lemma 5.5, it vanishes at 

P = {¢ e . l-I (",¢)= 0}. 

In particular, since 0 _< 7- < 1, I(foAa) is contilmous and takes the vahle 0 on P. 
Now we write the norm of this function in terres of a Riemann sure: 

Iii(-oÀoA)llLq,(b) lira [ ew Va I.b.(foA)(k-£)l:]I/q ' 

where Va denotes the volume of a cell of Aw. Moreover, Ca(x) = k«fo(kx)A,(kx) 
is supported in E and the relation .To(foA)(k-) = k-«grT(¢k)(A) is satisfied 

for ail A  Aw. Taking a = T[÷I + r/q, we obtain 
 =V/q' liln [ Z 
AEAw\P 

IZT(¢)()I4 /' 
ri io,/i«] ' 
aT+ 

since we know that for   P we get nothing. Finally, let us define  : D -* C 
by the relation ¢ = kA. The function  satisfies k(|l/'(x)) = k(x) for all 
IV  1/Va and is supported in k-E; hence we can understand ç as a central 
flnction on G. We can also say that, as a consequence of the well-known relation 

(4) A : exp_ Il (exp«-l), 

 has no singularities. Therefore Proposition 5.3 provides the following relation 
for some constant I3(G, q) depending on G and q: 

(5) 

III-(foA,)IIL«(,a) = (c,q)liln 
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On the other hand, since 9k can be seen as a central function on G, we can 
estinmte the norm of 9k on L°(G). By the "Weyl integration fornmla we get 

where the last inequality follows from (4). Now, under the change of variable y = kx 
and taking C(G, q) = (2rr) TIn+l Il/Val -1/°, we obtain 

Recall that supp(f0A6) c E; therefore the integral over kE (the domain of in- 
tegration after the change of variable) reduces to the saine integral over E. But 
a -r17+ I -r/q = O, and the product inside the integral is bounded over E, say by 
Ma. Then we can write 

(6) 

In summary, by (5) and (6), we know there exists a constant D(G, q), depending 

011 G and q, such that 

1C(G,q) = D(G,q) 

A 
IIG(f0A)llL«(a) < liminf IlOllc«(d)< . 

A 
Since fo is bounded, we easily obtain that foA6  L°(I)e), L-(foA6)  Lq'(Dâ) and 
K(G, q) > 0. Therefore we have round a family {k " k > 1} of central functions on 
(7 whose supports are eventually in/4 and such that their Hausdorff-Young quotient 
of exponent q is bounded below by a positive constant. This concludes the proof 
of Theorem 5.1 for compact semisimple simply connected Lie groups. 
If G is not simply connected, some extra comments bave to be made. We shall 
hot give complete proofs of any of them; the details are left to the reader. 
(i) Generalization (3) of formula (1) has no meaning here, but we can generalize 
formula (2) as 

1 
f(Tr) =  det W .T'T(fB6)(wt(A + 6) - 

This generalization provides a couple of results parallel to Lemmas 5.2 and 
5.5. Namely, 
 We have {Wt(+6)-6 : W e l/Va, A e ADW} = Aw\(P-6). 
The mapping (Il; )  Wa x ADW -* Wt( + 6) - 6  Aw \ (P - 6) is 
injective. 
 If f: G -+ C is central, then '(fBa)({) = 0 for all { e P - 6. 
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(ii) Proposition 5.3 is now replaced by the following identity, valid for central 

functions f  G -- C: 

I'T_(fB)(,X) 11/¢ 
II I(,+a)l «-= 

(iii) The bases of t) and 11 respectively that geuerate Aw and Lw with integer 
coeflîcients are no louger the basis of fundamental weights and its predual. 
In fact, the fundamental weights generate the weight lattice of the mfiversal 
covering group of G, which is a lattice containing Aw and strictly bigger 
than it. Therefore v need to define {HI, H2,..., H } and {Wl,W2 .... ,Wr } 
just as the bases of O and O respectively for which Lw and Aw have 
integer coefficients. Once we have clarified this point, we can define ç in 
the saine way and regard f0 as a bounded complex-valued function on 
supported in ç and symlnetric under the reflections that generate 
(iv) Let us recall that if   Aw, the function A5 is hot well-defined on T. But 
A, is originally defined on b, and  6 Av is hot an obstacle to working 
with A5 as a function defined on Oe. On the other hand, (ii) leads us 
to considcr, in the salue spirit as in the proof given for silnply colmected 
groups, the function 
Ç) (ç) = 1 a (f0Bs) (ç). 
Now, the reark about A6 shows that I(foB6)(ç) = I(foA6)(ç+). Hence 
we can proceed as before, expressing the norm of this function in 
as a Hiemann sum, but this rime we take the lattice Aw + instead of Aw: 
- VG 
IlIr(foB)llL«() = lim [  k 
(v) It is not difficult to check that (foAa)(k-lA) = k-«T(çBa)(£ - ), 
where çk is defined as above. Hence we get 
IIÇa)IL'() = V/q' lira [  T(çkB)(£)q' ,]/q' 
Finally, to estimate the norm of çk on Lq(G), we follow the smne argmnents. This 
completes the proof of Theorem 5.1 and, consequently, the proof of Theorem 3.1. 
Remark 5.6. Let {n : n  1} be a basis of neighborhoods of 1, and let 
(G,q) = inf sup  f central, f E Lq(G), supp(f) C H . 
This constant does hOt depend on the chosen basis, and Theorem 5.1 states that 
0 < (G, q)  1 for any 1  q  2 and any compact semisimple Lie group. However, 
it would be interesting to find the exact value of that constant. Sharp constants for 
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the Hausdorff-Young inequality were investigated iii [2], [3] and [13]. In the local 
case, if/q = ¥/ql/q/q'l/q' stands for the Babenko-Beckner constant, it is already 
known that /C('/I', q) = Bq. Andersson proved it for q' an even integer in [1], and 
Sj61in COlnpleted the proof, see [15]. Also it is obvious that IC(G, 1) =/C(G, 2) = 1 
for any compact group G. In the general case, a detailed look at the proof of 
Theorem 5.1 gives that the constant/C(G, q) is the suprennun of 

IWl'- II I(,,)1" V, l/q' liln 
k--'-)  

J I&¢/oA¢ç))l ¢ H I(°')12-q'd) llq' 
( î tET+ 

for 1 < q <_ 2, where the suprelmmi rulis over the family of flnictions fo : 191t --+ 
supported in E raid svnmietric under the refleetions generating the Weyl group 
of G. If 1Co(G,q) denotes the expression given above, then one easily gets that 
ICo(G, q) eqlmls 

Moreover, taking q = 2 and using the Plancherel tlieorem on compact groups, we 
see that VG = 1. The boundedness of this expression can be regarded as a weighted 
Hausdorff-Young ilmquality of Pitt type; see [4] for more on this topic. 

As we pointed out in the introduction, the growth of Ç_,lq(lP'(n), G) remains open 
for 1 <_ p < q < 2. We end this papcr with SOlne remarks about this problem. 

Remark 5.7. Iii Theol'eUl 3.1 we fotlnd ail extreuial fllllCtion n 
such t.hat 

II8,-,llcD(:>(a) > 1C(G,q)n,lp_,l q 
c'(t"(,),c) _> II,,ll<,,,,,,,(a> - 
Out functions qOl, qo,..., qo satisfied two crucial properties, nmnely, 
(Pl) the norln of (rr) on S °' does hot depend on k for anv 7r  ,, and 
dr 
(P2) qol, o,..., o bave pairwise disjoint supports on 
The idea was to compare the norms of , and , with n 1/p and rd/o respectively. 
To this end, properties (Pl) and (P2) were the conditions to be required, since they 
provided suitable simplifications for the original expressions of such norms. Now, if 
we replace IP(n) by IP'(n) in the relation above, we want to compare the norms of 
, and , with 71l/q' aud rt 1/p' respectively. Notice that llp- llq = llq' - llp'. 
For that, we require these other properties on qOl, qo ..... 
(P3) the absolute value Iqok(9)l does hot depend on k for any 9  G, and 
(P4) l, ,..., ç have pairwise disjoint supports on 
In the introduction we recalled that the growth of C(lP(n),G) and Cq(lP'(n),G) 
can be understood as dual problems with respect to the Fourier transforln operator. 
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Now, these properties justify this point. Assunfing properties (P3) and (P4), we 
get C(! p' (n),G) >_ K'(G,q,n)nl/q'-l/p', where K'(G,q.n) is given by 

Hence, ifwe define IC'(G, q) --- infr>l Kr(G, q, n), it. remains to sec that IC'(G, q) > O. 
We do hot know if this inequality hohls for any compact semisimple Lie group and 
any 1 _< q _< 2. 

Remark 5.8. We do hot know if properties (P3) and (P4) are compatible. However. 
given fo E L2(() continuous and any sequence of positive lmlllbers {gn " n _> 1} 
decreasing to 0. it is hot difficult to sec that there exists a system  = {ç,   >_ 1} 
of trigonometric polynomials on G satisfying the following three conditions: 
(1) The filnctions çl, ç2,..- have pairwise disjoint supI)orts on . 
(2) The estimate Içl < If01 + e holds in G. 
(3) The estinmte Iç,] >- ]fo]-¢, holds outsidc n, where p(,) --, 0 as  --, o« 
Remark 5.9. As is well known, C(lP'(n).G) = n I/p-1/q for any compact abelian 
group G. This equality follows l)y taking çl, ç2 .... , ç, to be a collection of n 
pairwise distinct characters. This motivates us to sec what happens when we 
consider the irreducible characters of a compact semisinlple Lie group. Let \x 
be the character of the irreducible rei)resentation 7rx, and consider the fllnction 
d r ,  ... d  (x,(g)), where /1,/2, A n are pairwise 
distinct donfinant weights and  = 1 - 2/q . Then we have 

On the other hand, applying consecutively the Weyl integration fornmla and the 
Weyl character fornmla, we get 

[AA+5(t)[P')q/P'dm(t)) 1/q 

However, these relations do hot provide optinml growth. For instance, in the sim- 
plest case (7 = SU(2) it can be checked that there exists a constant ICp.q. depending 
on p and q, such that 

1l 1/p'I ( U(2) II@"(g)llï"'('>dP(g))l/q >- 

Remark 5.10. If we trv to find out why our attempts to get optimal growth have 
failed, we need to revisit the proof of Theorem 3.1. The point is that we required 
the functions ç, ç2,--., ç, hot only to satisfy properties (P1) and (P2), but also 
to be translations of a colnmon fimction. This was essential in Section 3, and here 
the obstacle lies in the fact that we cannot take translations, since the dual object 
does hot have a group structure. This is the main difference froln the abelian case 
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where, since the dual object is a group, multiplication by a character in G becomes 
a transla.tion in the other side of the Fourier transform operator. 

Remark 5.11. The quantized Radenmcher system associated to a probability space 
(12, jU[, p), an index set E, and a family {do : er E E} of positive integers is defined 
by a collection T = {pe : ç -+ O(d«)}«ez of independent random orthogonal 
matrices, uniformly distributed on the orthogonal group O(do). In [9] we define 
the notions of T-type, T-cotype and strong T-cotype of an operator space E. 
Moreover, we show that 

Fourier type p :=> strong T-cotype p', 
Fourier cotype p' = T-type p. 

These implications allow us to work with the quantized Rademacher system, where 
other techniques are available to study the growth of C (1 p' (n), G). 

Remark 5.12. Of course, the growth of C(IP'(), G) is trivially optimal when we 
work with compact groups with infinitely lnany inequivalcnt irreducible represen- 
tations of the saine degree d0. The unitary groups U(n) are the simplest non- 
commutative examples of this degenerate case. Also, il is hot difiïcult to check that 
C(l p' (Il), G) - Il l/2-11p' by the Plancherel theorem for compact groups. 
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LEFT-DETERMINED MODEL CATEGORIES 
AND UNIVERSAL HOMOTOPY THEORIES 

J. ROSICK AND W. THOLEN 

ABSTRACT. We say that a model category is left-determined if the weak equiv- 
alences are generated (in a sense specified below) by the cofibrations. While 
the model category of simplicial sers is not lcft-determined, we show that its 
non-oriented variant, the category of symmetric simplicial sets (in the sense of 
Lawvere and Grandis) carries a natural left-determined model category struc- 
ture. This is used to give another and, as we believe simpler, proof of a recent 
result of D. Dugger about universal homotopy theories. 

1. INTRODUCTION 
Recall that a model category 1C is a complete and cocomplete category/C equipped 
with three classes of morphisms C, 14; and ', called cofibrations, weak equivalences 
and fibrations, such that 
(1) (C,'fq 14;) and (C fq 14;,') are weak factorization systems and 
(2) 14; is closed under retracts (in the category /C - of morphislns of/C) and 
has the 2-out-of-3 property 
(see [Q], [H], [Ho] or [AHRT2]). Model categories were introduced by D. Quillen 
to provide a foundation of homotopy theory. Here a weak factorization system 
is a pair (/, 7) of morphisms such that every morphism has a factorization as 
an £-morphism followed by an 7-morphism, and 7 = £0, £ = 137 where £° 
([]7) consists of all morphislns having the right (left) lifting property w.r.t. £ (7, 
respectively). The morphism I has a left lifting property with respect to a morphism 
r (or r has a right lifting property w.r.t, l) if in every comnmtative square 

A 
"U 

there exists a diagonal d  B -- C. 
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A model category is determined by any two of the three classes above. Clearly, 
C and 1AI determine 9 t- because 9 t- -= (C 13 l/V) I, and from C and 9 t- one obtains 
the morphisms of 1AI as composites g  f with f E []9 t- and g E C []. hl this paper 
we are interested in the model categories whose model structure is determined by 
its cofibrations only, and we therefore call them left-determined. For example, the 
model category SComp of simplicial complexes is left-deternfined while the model 
category Sirop of simplicial sets is hot lefl-determined. 
Sirop is, of course, the presheaf category Set A°P where A is the category of 
nonzero finite ordinals and order-preserviug nmps. F. \V. Lawvere IL] and M. Gran- 
dis [G] introduced symmetric simplicial sers as flmctors F °p -- Set where F is the 
category of nonzero finite cardinals (= finite sets) and arbitrary maps. \Ve will 
show that the category SSimp = Set r'» of symmetric simplicial sets is a left- 
determined model category, hlorcover, the nlodel categories SSimp and Simp are 
Quillen equivalent, i.e., they bave equivalent homotopy categories. 
c  Pt "°p 
D. Dugger [D] has recently shown that, for a small category A', mp is a 
uifiversal inodel category over ,¥. In particular, Sirop is a uifiversal model category 
over the (one-inorI)hism) category 1. We will give another proof of his result, by 
showing tha.t also a:mp serres as a universal model category over 2c'. Since 
SSimp is left-deteriifiimd, our proof is siinpler. 
The first author would like to ackilowledge his gratitude to Tibor Beke for in- 
troducing him to homotopy theory. Both authors acknowledge stinmlating conver- 
sations with him regarding the subject of this paper. 
Affer having completed this work we learned that the concept of a leff-determined 
model category was independently developed by J. H. Smith [SI, who used the terre 
minimal model category instead. He also observed that the usual model structure 
on simplicial sers fails to be left-determiimd. 

2. LEFT-DETERMINED hIODEL CATEGORIES 

Definition 2.1. A model category/C is left-determined if 1AI is the smallest class 
of morphisms satisfying the following conditions: 
(i) C r c_ W, 
(ii) ]42 is closed under retracts and satisfies the 2-out-of-3 property, 
(iii) C  W is stable under pushout and closed under transfinite composition. 

We will denote the smallest class of morphisms satisfying (i)-(iii) by lA]c. It 
has the property that l/Vc Ç 1AI for each model category/C having C as the class 
of cofibrations. Left-determined model categories are those for which 1AI = lA]c. 
Recall that C [] denotes the class of morphisms having the right lifting property 
w.r.t.C. Of course, C [] = 9 t- 13 1AI is the class of trivial fibrations. 
In general, given C, the first principal problem is whether C and l/Vc yield a 
model category. The next theorem gives an afiïrmative answer under an additional 
set-theoretic hypothesis, the Vopênka's Principle. (Subsequently, J. H. Snfith in- 
fornled us that he has been able to prove the theorem even absolutely, i.e., without 
any additional set-theoretic hypothesis.) Recall that Vopênka's Principle is a set- 
theoretic axiom implying the existence of very large cardinals (see [AR}). \Ve de- 
note by col(Z) the smallest class of morphisms containing Z, closed under retracts 
in comma-categories A\/C and satisfyillg (iii). The smallest class containing Z and 
satisfying (iii) is denoted by cell(Z) (see [AHRT1]). 
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Theorem 2.2. Let Z be a (small) set of morphisms in a locally presentable category 
K. Under Vopênka's principle, C = cof(Z) and 142 = 1/Vc yield a model category 
structure on . 

Pro@ According to the theorem of J. H. Smith (sec [B, 1.7]), it sutfices to show 
that 142c satisfies the solution set condition at 2-. This means that for every f E 2- 
there is a subset X I of 142 c such that every morphism f --* g, g E 142c, factorizes 
through some h  A" l. Since f\142c is a full subcategory of f\K and f\K is locally 
presentable (sec [AR, 1.57]), f\lNc has a small dense subcategory ,12 I provided that 
we asssume Vopênka's principle (sec [AR, 6.6]). Without any loss of generality, we 
may assume that ,12 I contains the initial object of f\142c provided that it exists. A 
morphism f --* g in f\ld2c is either initial in f\ld2 c and thus belongs to '¥1, or it 
factorizes through some morphism f --* h from A I. Hence A'i, f  2-, provide a 
sohltion set condition at Z. [] 

A model category is called cofibrantly generated ifC = col(2-) and Cfq)/V = cof(:7) 
for sets 2- and ,7. Following J. H. Smith, a model category/ is called combinatorial 
if it is cofibrantly generated and the category E is locally presentablc. The model 
categories from Theorem 2.2 are combinatorial. 
Left-deternfined model categories are, in some sense, related to left Bousfield 
localizations. Recall that, having model categories E and/2, a left Quillen functor 
H : K --* £ is a left adjoint functor preserving cofibrations and trivial cofibrations 
(i.e., elements of C fq l/V). Every left Quillen functor preserves weak equivalences 
between cofibrant objects (see [Ho]). An object A of a model category E is cofibrant 
if 0 --* A is a cofibration. 
A model category E is called functorial if both weak factorization systenls (C, 9rfq 
l/V) and (C fq l/V, 9 r) are functorial. This means that, for a weak factorization system 
(/2, 7), there is a functor F : E --* E and natural transformations ) : dom --* F 
and 0 : F -- cod such that f = 0I " )V is an (/2, 7)-factorization of a morphism 
f : A --* B; of course, dom(f) = A and cod(f) = B. This definition of a functorial 
weak factorization system is given in [RT] where its relation to fimctoriality in the 
sense of Hovey [Ho] is explained. Each combinatorial model category is fimctorial. 
In a functorial model category K we have a cofibrant replacement functor Q :  --*  
where 
0  Q(A)  A 
is a functorial weak factorization in (C,c çl 142). Then q : Q --* Idtc is a natural 
transformation. 
Let K be a model category and Z a class of morphisms of te. A left Bousfield 
localization of K w.r.t. Z is a model category structure K\Z on the category K 
such that 
(a) K\Z has the same cofibrations as K, 
(b) weak equivalences of tC\Z contain both the weak equivalences of K and the 
morphisms of Z and 
(c) each left Quillen functor H : K --* /2 such that H-Q sends Z-morphisms 
to weak equivalences is a left Quillen functor tC\Z --*/2 
(sec [H, 3.3.11). J. H. Smith proved that if K is a left proper combinatorial model 
category and Z is a set of morphisms, then a left Bousfield localization K\Z exists 
(sec [S]). (The model category is called left proper if every pushout of a weak 



3614 J. FtOSICK AND W. THOLEN 

equivalence along a cofibration is a weak equivalence.) As a consequence, we get 
t.he following result. 
Theorem 2.3. Let IC be a lefl proper, combinatorial model category and Z a class 
of morphisms of IC. Under Ibpênka's principle, a lefl Bousfield localization IC\Z 
exists. 
Pro@ We can express Z as a talion of an increasing chain of (small) subsets Z, 
indexed by ordinals. Let 142i denote thc class of weak equivalences iu the model 
category /C\Z (which exists by the rcsult of J. Smith). Then we have Wi C_ W) 
for i _< j: this follows from Idc " IC --, IC\Z2 being a left Quillen functor sending 
Z morphisms to weak equivaleuces. Hencc Q = Idc. Q " IC\Zi --, IC\Z) is a leff 
Quilleu flmctor. Let f  A  B be a weak equivalence from W and consider 

Qf 

Since fA, rB  C [, we bave 

r, Qf  kVj. Hence f  kVj. Put kV. 
iGOrd 

Then lA?. is closed under retracts and satisfies the 2-out-of-3 property. Analogously 
as iii Theoreln 2.2. Vopênka's principle guarantees that/C. C and lA?. form a model 
category/C\Z. 
Let H : /C --, £ be a left Quillen fimctor such that H  Q sends Z-morphisms 
to weak equivalences. Since H : /C\Z + £ is a left Quillen flmctor for each i, 
H :/C\Zi  £ is a left Quillen fimctor. Hence/C\Z is a left Bousfield localization 
of/ w.r.t.Z. [] 

Using [Atl] as well, C Casacuberta. D. Sceveneles and J. H. Smith [CSS] proved 
a related result saying that cohomological localizatious of simplicial sers exist under 
Vopênka's Priuciple. 
Remark 2.4. In mlalogy with the definition of a left-determined model category, we 
define 1/Vx, whcre X is a class of lnorphisnls in a model category/C, as the smallest 
class of norphisms satisfying 
(i) WU X ç Wx, 
(il) l/F.c, is closed under retracts and satisfies the 2-out-of-3 propertv, 
(iii) C fq 1/Vx is stable under pushout and closed under transfinite composition. 
Under Vopenka  principle, /C, C and I/V., form a model ca.tegory structure for 
each combinatorial model category/C. This model category structure is evidently 
/C\X, provided that /C\X exists (because ]/Vx is contained in the class of weak 
equivalences of 
Let/C be a cofibrantly generated model category and X a small category. Then 
there is a cofibrmltly generated model category structure on the functor category 
/C x° (see [H, 14.2.1]); this structure is called the Bousfield-Kan structure. To recall 
it we denote by 
evx " IC x°p -- IC 
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the evaluation fllnctor given by ev.(A) = A(X) and by 
Fx  K--, K "v°p 

its left adjoint given by 

.to(A,V) 
If 2- (resp., ,.7) is the set. of generating (resp., trivial) cofibrations in /C, then the 
Bousfield-Kan model structure has ç = [_J Fx(2-) as generating cofibrations 
XEob(X) 
raid 7 = U Fx(7) as gelwrating trivial cofibratkms. Then 
XEob(,Y) 
(a) p" A --,/3 is a weak equivalence in/C "v°p iff x " A(X) --, B(X) is a weak 
equivalence in K for each X in A', 
(b) p  A --/3 is a fibration in/Ç a'° iff çx " A(X) --/3(X) is a fibration in/C 
for each X in W. 
Consequently, trivial fil»rations are also lnorphisms in /Ç v°, whi«h are pointwise 
trivial fibrations in/Ç. 

Proposition 2.5. Let  be a cofibrantly generated, lefl-determined model cate9ory 
and A" a small category. Then K x° is a left-determined model category. 

Pro@ Let 2" (resp., ) be the set of generating (resp., trivial) cofibrations in 
and let  be the set of weak equivalences iii ]Ç X°p . "V have l&co(Z) C 142. Let 
u, G l/V. Then u, = f "9 where f G and 9 G cof(F). We have f G Wc«(Z ). To 
prove that 9  Wc«(Z), that is, w  Wc«(Z ), it suffices to show that , ç Wcof(). 
But this follows froln J ç lA;cofCr) and the fact that Fx preserve colinfits. [] 

3. SYMMETRIC SIMPLICIAL SETS 
A trivial exmnple of a left-determined inodel cat.egory is the categoI'y Set of sets, 
with C the class of ail monolnorphislnS, and with 142 the class of the morphisms 
between nonempty sers and the identity morphism on 0. To give a nontrivial 
exmnple we recall t.hat a sirnplicial complex is a set X equipped with a set .' of 
nonempty finite subset.s of X such that 
(a) {x} G X for each x G X, 
(b) A e X, O C B ç A=> B e,Y. 
Elements A  X with lAI =  + 1 are called (non-degenerated) -simplices. For 
n = 0, 1 and 2 we speak about vertices, edges aud trïangles, respectively. If lAI <_ 2 
for each A e X, then (X, X) is called a (non-oriented) graph (with loops). Mor- 
phisms of complexes (X,X) -- (]'\ç) are maps h  X -- Y with h(X) C ç. 
We denote the category of simplicial complexes by SComp. We will show that 
C = Mono yields a left-determined lnodel category structure on SComp. But the 
disadvantage of simplicial complexes is that each simplex is uniquely determined 
by its vertices, which makes colimits in SComp bad. This led S. Eilenberg and 
B. Zilber [EZ] to introduce complete semisimplicial complexes, which later were 
renamed as simplicial sets, and which are oriented. Surprisingly, non-oriented sim- 
plicial sets were introduced only recently by F. W. Lawvere [L] and M. Grandis [G]; 
they are called symmetric simplicial sets. Their position to simplicial complexes 
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is the saine as the position of nultigraphs to graphs in graph theory (one admits 
multiple edges). 
Definition 3.1. Let F denote the eategory of nonzero finite cardinals (and ail 
nmps). A symmetric simplicial set is by definition a functor F °p  Set. The 
category Set F°P of symmetric simplieial sers will be denoted by SSimp. 
We will recall the basic properties of symmetric simplicial sers (see [G]). We 
have the Yoncda embedding 
Y : F -- SSimp. 
Its values An_ 1 = ]rr(T/) are in fart simplicial complexes, which yields the functor 
F --, SCornp. 
The Yoneda embedding Y extends along this functor to the full embedding 
G : SComp --, SSimp 
sending a simplicial colnplex (X, X') to the functor A : F °p --, Set given by A(n) = 
{S E X' I]SI <_ u - 1 }. In what follows we will identify a simplicial complex (X, 
with its image under G. Hence SComp will be considered as a full subcategory of 
SSimp. 
We bave the functor U : SSimp  Set given by precomposition with 1 --* F °p 
(sending the object of 1 to 1). We can view U(A) as the set of vertices of a 
symmetric simplicial set A and the whole A as the set U(A) equipped with n- 
simplices corresponding to morphisms A, --, A. We will use the notation A = 
(UA, .4) where .4 is the set of simplices of A. For instance, A, bas ail nonempty 
subsets of {0, 1 .... , n} as simplices. But note that the functor U is hot faithful. 
The embedding A --, F induces the faithflll functor 

It has a left adjoint 

H " SSimp -- Simp. 

L  Simp  SSimp 
sending each simplicial set to its symmetrization. There is also a right adjoint 
R  Simp -- SSimp. 
Let ôA n be the boundary of A for n > O, i.e., U(ôAr) -----  + 1, and simplices 
of ôA, are ail nonempty subsets of {0, 1 .... ,n} distinct from {0, 1,... ,n}. Let 
in " Ô/kn  An, n > 0, be t, he embeddings. Let 2" = {i n I n >_ 0} where 
i0 " 0  A0. 
In what follows, the class of ail monomorphisms of SSimp is denoted by Mono. 
Lemma 3.2. cof(Z) = 11Ioo. 
The proof is the saine as for simplicial sers. 
Given 0 _< k _< n, the k-horn A is the simplicial complex whose simplices are 
ail subsets  ¢ S C {0,1 .... ,} distinct from {0 .... ,k- 1,k + 1,...,n}. Let, ,.7 
be the set of inclusions 
j -/x -/x, .>0. 
Lemma 3.3. j, E WM,  Mono for each n > O. 
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Pro@ Evidently, each morphism sn : An -- Ao, n > 0, belongs to Mono [. There- 
fore, by 2.1 (i) and (ii), cach morphism u : Ao -- An belongs to 14;Mono and, 
cousequently, to 14;^Zono f3 Alono. Hence il E VMono f3 Alono. 
Assume that jl,..., j.n  W^ono n Mono. Consider thc pushout 

An + Ao  > An 

where uï(O) = 0 and Pn is induced by 

and u n° . Ao -- An (again u°(o) = 0). 

Since idA, +uï  WMono  Mono, we have gn  WMono  Mono (by 2.1 (iii)). Pn 
is the simplicial complex given by attaching an edge at the vertex 0. By successive 
use of j2,... ,jn, we fill horns to simplices. This is done via pushouts, starting with 

1 I 

where h sends one edge of A ° to the attached edge and the other edge to an edge 
of An. Doing this for all edges of A containing 0, we start to fill by using J3, etc. 
o 
At the end we obtain A,+l and a morphism 

gn I 0 
qn. An-- Pn---, P---,...---, A,+l 

which belongs to W^lono f Alono. Since, in the diagram 

q' 0 
) /n-{- 1 

A o 

we have sn G Wtono, we get tn G VVMono. Since, in the diagram 

we have tn, 8n+l 
Alono. 

0 
/n-{-1 ) /n-{-1 
Ao 

Ç Wllono, we get jn+l  W^,ono. 

Hence jn+l  W^lono çl 
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Theorem 3.4. SSimp zs a left-determined model category wzth C = ]llono and 
 = J[]. 
Pro@ (Mono, Mollo c) is a weak factorization system (see lB] or [AHRT2]); anal- 
ogously for (cof(7), 71). To prove the result it sutïices to show that 
WMono a Moo = cof(J) 
(cf. [BI). Following Lemma 3.3, we have 
cof(J) ç WMoo  Mo. 
The opposite inclusion will follow flore properties of the adjunction L q H between 
symmetric simplicial sets and simplicial sets. 
Since L preserves monomorphisms, H preserves trivial fibrations, i.e., 
H(loo ) ç W, 
where W denotes the cls of weak equivalences of simplicial sets. Since H preserves 
nlOilOlllorphislns  well, we have 
H(WMoo  Mo'o) ç W  Moo* = cof(J*) 
where Mw* denotes the monomorphisms in Sirop and ff* is the generating set of 
horns in simplicial sets (cf. [Ho]). Since L(fl*) = ff, we have L(cof(fl*)) ç «of(J). 
Consequently, 
LH(Wtoo  Mo) ç col(ff). 
The functor H sends a symmetric simplicial set A = (UA, A) to the simplicial 
set HA having  (oriented) simplices all possible orientations of simplices from A. 
The functor L then produces from eh orientation a non-oriented simplex in LHA. 
Hence LH multiplies eh non-degenerated n-simplex in A n times. By sending 
each simplex in A to its standard orientation, we get a natural transfornmtion 
0 " Id  LH that splits the adjunction counit e  LH  Id. Hence each morphism 
f in SSimp is a retrt of LH(f). Consequently, 
WMoo  Mo ç col(ff). 
Remark 3.5. Both L and H are left Quillen functors. Moreover, L  H is a Quillen 
equivalence. Following [Ho, 1.3.13], this amounts to showing that 
X  HLX « , H(LX)I 
is a weak equivalence for each simplicial set X (where ç is the adjunction unit and 
rx " LX  (LX)I is a fibrant replacement) and that 
ev  LHY  Y 
is a weak equivalence for each fibrant symmetric simplicial set Y. But çx" is a 
trivial cofibration because it is given by completing horns to simplices, and HrLx 
is a triviM cofibration too, because H is a left Quillen functor. That e- is a trivial 
fibration follows from its description given in the proof above. 
As a consequence we obtain that Sirop and SSimp have equivalent honotopy 
categories. 
Remark 3.6. The model category Sirop is not le-determined. To prove this we 
consider the class of morphisms f " A  B such that one of the following 
possibilities occurs (U* denotes the underlying functor Sirop  Set ): 
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(a) there are vertices bi  U'B, b2  U*B - (U'f)(U'A), an edge e in B from 
bi to b2, but no edge in B from b2 to bi; 
(b) there are vertices al, a2  U*A and an edge e in A from al to a2 8uch 
that there is no edge in A from a2 to al, but there is an edge in B from 
U*(f)(a2) to U*(f)(a); 
(c) there are vertices al, a2  U*A and an edge e in B from U*(f)(a) to 
U*(f)(a2), but there is no edge in B from U*(f)(a2) to U*(f)(a) and no 
edge in A from al to a2. 

Since (the oriented) horn jï  Aï  A 1 belongs to ,V, it suflïces to show that 
X çkV^ono- = 0. But .Vç Mono[] = 0 and IlO eleinent of A' can arise by operations 
2.1 (ii) and (iii) froiu noI'phisnls hot belonging to ,t'. 

Remark 3.7. SComp is a leff-determined model category with C = Mono and 
9  = ff[]. In fact, both in, n _> 0 and j, n > 0 are morphisms of simplicial 
complexes. Hence the result follows froin Theorem 3.4. 

Corollary 3.8. For each small category X the functor category SSimp r is a lefl- 
determined model category (with the Bousfield-Kan model category structure). 

The proof follows from Theorem 3.4 and Proposition 2.5. 

4. UNIVERSAL MODEL CATEGORIES 
We will show that SSimp x° is a universal model catcgory over ,¥ in the sense 
of D. Dugger [D] for each small category X. hl particular, SSimp is a universal 
model category over the one-morphism category. We will denote by 
)'*  oe  lmp 

the composition 

2(  Set x°"Dx « (Set x°' F°" where Dx is a left adjoint to the underlying functor 
Ux " (Set x°' F°"--- Set x°" given by evaluation at 1, i.e., Ux = eVl. Of course, we use the identifications 
(Set F°) x°"  Set (FxX)°  (Set x°) F . 
- .op 
Objects of mp may be called symmetric simplicial presheaves; then Dx(A) 
is the discrete symmetric simplicial presheaf over A. XVe also have the neda 
embedding 
)" F x X  SSimp "v' 
and we will use the notation 
A,x = Y(n + 1, X) 
for n  0 and X  X. It is easy to see t hat 
A,x = 5(A) 
 op 
where Fx " SSimp  SSlmp is a left adjoint to the evaluation functor evx-. 
We will also denote 
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Theorem 4.1. Let K be a functorial model category, 2( a small category and 
H  2(  K a functor such that ail objects HX, X E 2( are cofibrant. Then 
there is a left Quillen functor H* " 
 SS,mp  K such that H*  Y* = H. 

Pro@ Let F be the full subcategory of F consisting of cardinals 0 < k _< n + 1. 

We get the induced inclusions 
 
SSunp n 
vhere SSimpn = SetFP (SSunp" 

ç SSimp x° 
- SSunp is given by SSimp, - 

SSimp, which is induced by the functor F  F  SSimp). In particular, 
SSimp v°  Set x° is the category of discrete symmetric simplicial presheaves. 
Since K: is cocomplete and Set x° is a free cocompletion of 2( (see [Ait, 1.45]), H 
extends to a colimit preserving functor 

H  SSimpô ° -- K: 

such that H Y = H. 
Assume that we have the fimctor 

.  2d°P 
H,  SSmp, -- K 
extending H_. Since 0An+Lx, A0.x belong to SSimp °" for X E 2(, we can 
define H+(A+.x) by the functorial (cofibration, trivial fibration) factorization 

H(OA+Lx) «"+"" * 
of the morphism H((p))  H(O&+,x)  H(&,x) where p+ - OA+  
0- To get an extension H+ of H, below we will define 
(a) H+(f) for f = Y(f,fz) " A+,x  A+Lv where f  {0.1 ..... 
n + 1 }  {0.1 ..... n + 1 } is a bijection, 
(b) H.+l(t ) for t = Y(tl,tz)  Am.x  A+,- where m  n, 
and 
(c) H+ 1 for u = Y(Ul, u)  A+l,X  A,v where m  n. 
(a) fl induces the isomorphisms   +1  +1,  " 0+  0+1 
and f induces a natural transformation çl " Fx  Fv. Hence f induces the 
homomorphism 0f  0A+,x  0A+Lv. We define H+l(f) by the functorial 
filling 

H(OAn+l,X) 

Cn+l ,X Tn+I,X 

H (Of) 

H+I 

H (OAn+l,I/) C'n-t-l.Y ) H+I(/n+I,Y) 
(b) Since t factorizes through in+l,Y, i.e., 
we put H+I(t ) = c,+,v. H(t'). 

in+l ,Y 

> /n+l,I/, 

H, ((oI2)'o ) 

> H(A»-) 
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(c) To define H+ l(u) for each u, it suffices to do this for the retraction u ° of 
A+l,X to one of its faces A,,x. hl this case, we take H+l(U °) given by the lifting 
property 

> H+I(A.+I,X) 
"Cn÷ I,X 

H;(/X.,x) » H;(/%,x) 
H:(s,.x) 

where smx = Y(s, idx) and s   + 1 -- 1. 
To prove that H+ 1 is a functor, it suffices to consider the following cases: 
(1) In 

H+I (t) 

) H÷l(An÷l,Y ) 

"f n-I-1, Y 

H: (Of) 

H+l(f) 

:((z)ao) 

H(OAn-I-l,Z) 

) H_l_l(An-I-l,Z ) )//(AO,Z) 
Cn÷ l,Z "Cn÷ I,Z 

we have u °  
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We have dcfilmd H,+ Oll the image of the Yoneda embedding 
}-,  F, x A'  Set (F"xg 
= llllpn 
Since this is a free cocompletion of F x A', we obtain a colimit-preserving flnctor 
Hn+ 1 " allnpn  K extending H* 
We have constructed an increasing chain of colimit-preserving flmctors H  
S. 
Sllnp;  K for  = O. 1,..., whi«h yields a colinit-preserving functor 
H*  SSimp 
 " "VP H* 
with the restriction H, on lmp, . 5Ioreover, is a left adjoint functor 
(see (a) in the proof of 1.45 in lA,l). In particular, H'Y* : HY* : H. It 
remains to be proved that H* preserves cofibrations and trivial cofibrations. Since 
0A,+l.X is a colimit of 0,,,¥, m  zt, by (b) of the construction we get that 
H*(n+l,X) : cn+.x. Hence H" preserves cofibrations (using Lemma 3.2 and the 
fact that H* preserves colimits). Since S,+l,X : sn.x" u °, by (c) of the construction 
we get that H*(s,+..x-) : rn+Lx. Hençe H*(s,+l.X) is a weak equivalence 
for n > 0 and X  X. Since s,,.x  f = s,.r fir each morphism f  A..x  Am. , 
H*(f) is a weak equivalen«e in  as well. 
H*Ao.x  H*At.x is a weak equivah'uce. Since it is also a cofibration, it is a 
trivial cofibration. Assume that H*F¥(j) ..... H*Fx(j,) are trivial cofibrations 
in Ç. In the notation of Lemma 3.3, we get that H*F(gn) is a trivial cofibration 
in  because uï : j. Since by assumption H*Fv(j),...,H*Ç(j,) are trivial 
cofibrations, H*Çv(qn) is a trivial cofibration too. Therefore H*Çv(t,) is a weak 
equivalence in  and t.hus H*Fx(j+) is a weak equivalence too. Since it is 
cofibration, it is a trivial cofibration. 

Remark 4.2. Let/C be a flmctorial model category, X a small category and H  X -+ 
/C an arbitrary flmctor. Then there is a left Quillen flmctor H*  SSimp 'v°p -+/C 
and a natural transformation "ï  H'Y* ---, H v«hich is a pointwise trivial fibration 
in K. 
III fact, let Ho " X -+ K be the composition Q- H where Q  /C -- /C, is the 
(functorial) cofibrant replacelnent ftmctor and 7 = qH  Ho -- H the corresponding 
natural transforlnation. Then 7 is a pointwise trivial fibration in /C, (because q  
Q -- Idtc is). If we start the construction of the fimctor H* with Ho (i.e., H-}'* = 
Ho), we get the resnlt. 
Corollary 4.3. Let 1C be a functorial model category and Iç an object in K. 
Then there is a lefl Quillen functor H* " SSimp -- /C, and a trivial fibration 
"y" H*(A0) --> K. 
This is a special ca.se of Theorem 4.1 (for Xa one-morphism category). 
Remark 4.4. Again if K is cofibrant, then H*(A0) = K. 
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THE COMBINATORIAL RIGIDITY CONJECTURE IS FALSE 
FOR CUBIC POLYNOMIALS 

CHRISTIAN HENRIKSEN 

ABSTRACT. We show that there exist two cubic polynomials with connected 
Julia sets which are combinatorially equivalent but hot topologically conjugate 
on their Julia sets. This disproves a conjecture by McMullen from 1995. 

INTRODUCTION AND RESULT 
Let T)d : {z d --}- ad_2 zd-2 --}- ... --}- t0}  (d-I be the space of monic centered 
polynonfials of degree d > 1. 
Our object is to show that there exists a cubic polynomial f  73 that is not 
Colubinatorially rigid. To specify what this ineans, we need to introduce SOlne no- 
tation and results froln the theory of hololnorphic dynalnics, and, lnore specifically, 
the dynamics of polynomials. We will assume that the reader has SOlne knowledge 
of the theory of iteration of polynomials in one variable. There are several introduc- 
tory books on holomorphic dynalnics, such as [lkI1]; also Lyubich [L1] has recently 
published a survey article. 
The filled Julia set K(f) of f  7d is the set of values of z such that the orbit 
z,f(z),f o f(z) = f2(z),.., is bounded. The boundary of this set J(f) = OIx'(f) 
is called the Julia set of f, and it coincides with the set of points that adroit no 
neighborhood restricted to which the iterates id, f, f2 .... form a norlnal fanfily. 
We denote by Cd the connectedness locus of the degree d monic centered poly- 
nomials: 
Cd = {f  T)d [ tç(f) is conlmcted}. 
The connectedness locus of the quadratic polynomials, C2, is the Mandelbrot set, 
and we also denoted it by 
Given f  Cd, there exists a unique conformal isomorphism 
that conjugates f to z  z d and satisfies er(z) = z ÷ O(1/z). The conformal 
isomorphism is called a BSttcher coordinate. 
Using the BSttcher coordinate, we can define the notion of the dynamical ray 
Rf(O) of angle 0  I/Z as the preimage ¢)-{exp(ç ÷ 2irO) I ç > 0} of a radial 
segment. For a polynomial f  7), we define the potential Gf  C. \ K(f) -- I by 
1 
Gf(z)= lira log Ifl(z)l 
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Then, when K(f) is commcted. G i = log I¢i[, and a dynamical ray is parallel to 
the vector field grad G i. This allows us to generalize the notion of a dynamical ray. 
Indeed, for an arbitrary polynomial f E 79a, the dvnamical ray RI(0 ) is the curve 
that is tangent to grad G i and tangent to the segment {r exp(2irrO) [ r > 0} near 
infinitv. 
Following a dvnamical ray down potential, two things nfight happen. Either the 
roEv bits a critical point of G' i and we say it bifurcates, or it accumulates on the 
Julia set. The first behaviour only takes place when G' i has critical points, which is 
equivalent to f having critical points that are attracted to infinity and to ffi being 
discomected. 
Suppose the ray RI(0 ) does hot biflrcate. Consider RI(0 ) \ RI(0 ). This contin- 
uum is called the accumulation set of the rav. If it çonsists of only one point, we 
say that the ray lads. In the commçled case, f E Ca, that means that the linfit 
z0 = limo\0 i-1 o exp(r/+ 2irr() exists. 
Rays of rational angle are special, in that thcy always land when f  Ca " 
Proposition 1. Let f  79 ard 0  Q/Z. Thcn the ray BI(O ) either bifurcateô or 
lands af a (pre)periodic point % In the latter case, 7 is periodic if O is periodic, and 
preperiodic if 0 is preperiodic; it is either repelling or parabohc. [] 
Thc proposition is proved in the Orsay Notes (Proposition 2 of exposé 8 in 
[r]). 
This proposition allows us to distinguish the two fixed points of a quadratic 
polynomial P«  z -, z 2 + c when c /I \ { 1/4} (for c = 1/4. P« has one fixed point 
aud it is of nmltiplicity 2). Indeed. the ray Rp(O) is fixed bv P« and then must 
land at a fixed point. This fixed point is called the/3-fixed point, and the other 
fixed point is called the a-fixed point. 
Definition 1. Following McMullen ([Mcl]) we define the rational lamination AQ(f) 
C Q/Z × Q/Z of f  Cd to be the equivalence relation under which two rational 
angles 0  and 0  are equivalent if and only if the two dvnamical rays R.f(0 ) and 
RI(O ) land at the saine point. 
Definition 2. A polynomial f  Cd with no indifferent cycles is said to be combina- 
torially rigid if for every g  Cd with no indifferent cycles such that AQ(f) 
the composition of B6ttcher coordinates ¢  ¢I extends to a quasiconformal 
homeomorphism on the whole Riemann sphere. 
Note that we do hot require the extension to be a conjugacy on the sphere; 
however, by continuity, it will be a conjugacy on the Julia set J(f). 
The following statement has been conjectured: 
Every polynomial f  Cd without indifferent periodic cycles is combinatorially 
rigid. (See [Mcl].) 
This conjecture can be viewed as a series of conjectures, one for each value of 
d= 2,3,... 
If the conjecture is true for quadratic polynomials (d = 2), then it would imply 
that the Mandelbrot set. M is locally connected [Sch]. Local connectivity of M 
is one of the most important questions in the study of the dynamics of quadratic 
polynomials. It has been verified for most types of parameters in the boundary 
of M. A smumary of known results on this question is given by Lyubich in [L2], 
Appendix 13. 
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We shall show tlmt the conjecture is false in degree 3, by finding two cubic 
polynomials with colmected Julia set.s, with no indifferent cycles aDd the saine 
rational lamination, but which rail tobe even topologically conjugate on their Julia 
set.s. 
A sketch of the proof follows: From work of Sçrensen [Soi it follows that there 
is an infinitely renormalizable quadratic polynonfial Pc : z - z2+ c whose 
Julia set contains a subcontimmm without periodic or preperiodic points. Choose 
a quadratic polynomial Q whose critical point eventually lands at the -fixed point. 
Let fo be the result of intertwining Q with P¢ at their -fixed points, using the 
intertwining surgery construction of Epstein and Yampolsky [EY]. 
The cubic polynomial fo has a quadratic-like restriction fo : U  -- U that is 
hybrid equivalent to P¢. 
There is a neighborhood A of fo such that for each fx there are disks U[, Ux for 
which the restriction fx : U[ -- Ux is quadratic-like. This tan be accomplished so 
that the family (f),  U, -- U),) is an onalytic family in the sense of [DH2]. Using 
results in [DH2], we can suppose that the set of parameters for which the corre- 
sponding quadratic-like nmppings ail are hvbrid equivah,nt to P« locally forms a 
codimensioll-one submmlifi»ld of the space of cubic imlynomials. 
For paralneters t in this submanifold, one critical point wl(t) belongs to this 
quadratic-like restriction, whereas the other w2(t) does hot. Using holomorphic 
motion arguments, we sec that there are copies of (part of) the Julia set of P 
showing up in the set of t-values for which ft has connected .hllia set. Loosely 
speaking, the position of the parameter t in such a parmneter copy corresponds to 
the position of an iterate of w2(t) iii a copy of Iç(P), in the dynamical plane. So 
by varying t we tan slide an iterate of w2(t) around in a continuum without periodic 
or preperiodic points. It follows that no rational ray bifurcates. However, we shall 
see that the dynamics on the 3ulia sets do change. 

1. A SPECIAL QUADRATIC POLYNOMIAL 

In this section we will find a quadratic polynomial that does hOt have indifferent 
cycles and whose Julia set contains a continuum with no periodic or preperiodic 
cycles. 

Polynomial-like mappings and renormalization. 

Definition 3. If U'. U are topological disks with U' compactly contained in U and 
f : U' -- U is a holomorphic, ramified covering of degree d > 1, then f : U' -- U is 
called a polynomial-like lnap of degree d. 
The notion of polynomial-like nmppings was introduced by Douady and Hubbard 
(see [DH2]) and is extremely useful. When the degree of a polynomial-like Inap is 
two, we call it quadratic-like. As the naine suggests, a quadratic-like map behaves 
qualitatively like a quadratic polynonfial. 
Analogously to the definitions for polynomials, we define the filled Julia set of 
a polynomial-like map f as the set of points that do not escape under iteration: 
K(f) = {z  U' I fl(z)  U',f2(z)  U',...}, and the Julia set J(f) of f as 
its boundary: J(f) = cgIç(f). Two polynomial-like mappings f, g are called hybrid 
equivalent if there exists a quasiconformal mapping ç that naps a neighborhood of 
K(f) onto a neighborhood of Iç(g), conjugating f to g and satis-ing c¢ _ ôe/ô _ 
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0 ahnost everywhere on K(f). This clearly defines an equivalence relation, whose 
equivalence classes are called hybrid classes. 
That quadratic-like mappings behave like quadratic polynomials follows from 
Douady and Hubbard's Straightening Theorem, which for quadratic-like mappings 
boils down to: 
Straightening Theorem. A quadratic-like map f is hybrid equivalent to a qua- 
dratic polynomiai. If Iç(f) is connecte& f is hybrid equivalent to a unique monic 
centered polynomial z -+ z 2 + \ (f). [] 
It may happcn that an itcratc of a quadratic polynomial in some region behaves 
like a quadraticqike map. 
Definition 4. A quadratic polynomial P : z - z 2 + c is called n-renormalizable 
for an intcger n > 1. if thcre exists a restriction P' : U' --, U that is quadratic- 
like with connected Julia set and 0 E U'. The restriction P' : U' -- U is called 
a renormalization of P', and the set of positive integers n > 1 such that P is 
n-rcnormalizable is callcd the levels of renormalization of P and is denoted T(P). 
If this sct is noncmpty, P is called renormalizable, md if it is infinite, P is called 
infinitely renormalizable. 
In [Mc2] McMullen shows: 
Theorem 2 (Uniqueness of renormalization). Any two renormalizations of pn 
have the saine filled Julia set. [] 
An infinitely renormalizable quadratic polynomial. If n E T(P), we denote 
by h-,(P) the uniquely determined filled Julia set of P : U' - U, and we set 
&() = 0(). 
In [Soi, Sorensen proves the following result: 
Theorem 3. There is an infinitely renormalizable quadratic polynomial P¢¢ : z - 
z2 +c with the foliowing properties: There is a sequence of integers ni < n2  ..- 
such that each nk  Tt(Pc¢¢), Ix'n« D h'n«+ and the continuum 
is nontrivial (i.e., contains more than one point). [] 
A proof can also be round in [bi2]. 
Let P« and J be as given by Theorem 3. Since J is obtained as an inter- 
section of nested full contimm, J is a flfll continuum. 
By the following theorem of McMullen, J contains no periodic or preperiodic 
points. 
Theorem 4. If a quadratic polynomial P is inflnitely renormalizable, then: 
(1) all periodic cycles of P are repeiling; 
(2) the fiiled Julia set K(P) bas no interior; and 
(3) /f R C (P) is inJïnite, then the continuum Ç]n K,(P) contains no pe- 
riodic or preperiodic points. 
Proof. This is an immediate consequence of [Mc2], Theorems 8.1 and 7.8. [] 
Arbitrarily small copies of J accumulate on the/3-fixed point of P: 
Proposition 5. Given any  > 0, there are an integer n > 0 and a topological disk 
contained in {Iz -/31 < } that Pcn¢ maps univalently onto a neighborhood of Job. 
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g 

FIGURE 1. The inverse branch g used in the proof of Proposition 5. 
Due to inherent difficulties in drawing the filled Julia set K(Pc), 
a filled Julia set for another parameter is drawn. 

Proof. By Theorem 4, the c-fixed point, c, of Pc is repelling, since Pc is infinitely 
renormalizable. By a result of Douady and Yoccoz, k rational rays land at c for 
some k > 0. The ray of angle 0 ° lands at the/3-fixed point, /3, and it follows that 
k > 1. The region W = {z I Gp« < 1} is a Jordan domain. The k rational rays 
landing at c cut this region into k pieces Vo, V,..., Vk-, which we enumerate 
cyclically with respect to c in such a way that the critical point 0 is contained 
in Vo. The preimage W' of W is cut into 2k - 1 regions V, V,..., Vk_ 2 by the 
rational rays landing at c and -c. We can assume 0 E V. Each piece V[ i =/: 0, 
is mapped properly of degree one to a piece V«(i), and V is mapped properly of 
degree 2 to a piece V«(0). Since none of the rational rays landing at c are fixed, 
Vo(0) does not coincide with Vo. It follows that Vo has two preimages: one V" that 
is compactly contained in Vo, and the symmetric one -V". Since 0 is contained in 
Jm and Jm contains neither a nor -c, we get that Jm is contained in Vo and does 
not intersect V". 
Now consider the inverse branch g = Pc- : Vo - V" (see Figure 1). This uni- 
valent map is a strong contraction with respect to the hyperbolic metric on Vo and 
necessarily contains a fixed point, which then must be/3. Now Vo, g(Vo), g2(V0),,   
is a sequence of topological disks, each containing/3 in its closure, whose diameters 
tend towards zero. Finally, Pcn maps gn(Vo) univalently onto Vo. [] 

2. CUBIC POLYNOMIALS 

In this section we consider the space of cubic polynomials, and see that deformed 
copies of (parts of) quadratic Julia sets show up both in dynamical planes and in 
parameter slices. 
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We parametrize the space of cubic monic centered polynomials by C 2, letting 
fa(Z) = Z 3 -- 3a2z + b, a = (a, b) E C 2. This is a four-to-one covering of the cubic 
polynomials modulo affine conjugacy, ramified over {ab : 0}. The two critical 
points wl : col(a) : a and w2 : co2(a) = -a are then global holomorphic functions 
of the pa.rameter a. 
Define Q C C 2 to be the set of parmneters for which fa can be restricted 
to a quadratic-like mapping fa : U' --* U with w E U'. We say that Q is the 
set of parameters such that fa is co-renormalizable, and we call fa : U' --* U an 
c-renorlnalization of fa- Similarly, dcnote by Q2 the set of parameters for which 
fa is co2-renormalizable. Notice that a cubic polynomial being wi-renormalizable 
is quite different than a quadratic polynomial being renormalizable, since in the 
former case we are ouly considering restrictions of the cubic map, whereas in the 
lattcr case we only consider restrictions of some nth iterate, n > 1. (Also, this 
terminology is hot completely standard, in the sense that it is usuallv required 
that the filled .]ulia set of the renormalization be connected for a mapping to be 
called renornmlizable, tlowever, in this context we find it convenient to drop that 
requirement for a,,i-renormalizations.) 
An col-renormalization (or an c2-renormalization) of fa has the following prop- 
erties, independent of the chok'e of domains: 

Proposition 6. Suppose fao is -renormalizable, and let fao " U --* U and fao " 
U --* U2 denote twowl-renormaIizations. Then/x'(/ao : U; --* U1) = /x'(/ao : U 
U2). Furthermore, if K(fao " U --* U) is connecte& tben both renormalizations 
are hybrid equivalent to the saine quadratic polynomial z  z 2 + \(a0). 

Proof. The result follows readily flore [Mc2] (Theorem 5.11) and from the Straight- 
ening Theorem. [] 

A one-dimensional submanifold of the cubic polynomials. Let KTi C Qi, 
i = 1,2, denote the subset of parameters a such that an wi-renormalization of fa 
has connected Julia set. 
It follows from Proposition 6 that for a given point a  KT, there is a unique 
monic centered polynomial z  z 2 + à:i(a) that is hybrid-equivalent to an 
renorlnalization of fa- This determines mappings \i of KTi into the Mandelbrot set 
M. 

Theorem 7. There exists a parameter ao G 1(71 satisfying: 
 (ao) = c (where c is the parameter whose existence is guaranteed by 
Theorem 3); 
 an iterate fâo(w2(a0)) is equaI fo the fixed point offao correspondmg fo the 
/3-fixed point of z  z 2 + c; 
 the set -(coe) is a one-dimensional subrnanzfoId of  IocalIy at a0. 
The first step in proving Theorem 7 is to establish that :k[-(coe) is an analytic 
set in Q. The definition of an analytic set is as follows. 

Definition 5. Suppose D is an open set in C . The set A is called analytic in 
D if for every z  D there exist an open neighborhood U of z in D and a finite 
number of holomorphic functions fi : U - C, i = 1 ..... k, such that A  U = {z  
U I fl(z ) .... =fk(z) :0}. 
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We will give a few properties of analytic sets. Proofs can be found in books on 
several complex variables, such as [KK] and [GF]. 
Let a0 E A. If there are flmctions ft .... , le, defined on a neighborhood U of ao 
such that A  U = {z  U  f(z) .... = f(z) = 0} and 
then a0 is called a regular point or a nmnifold point. By the Implicit Function 
Theorem, A is locally biholomorphically equivalent to a donmin in 
define the dimension of A at a to be dima. A = n - k. A point in A that is hOt 
regular is called singular, and we denote the singular points of 
Proposition 8. Let A be an a'nalytic set i'n the open set U C C n. Then the set 
of singular points S(A) is analytic in U, and the set of regular points A  S(A) is 
dense in A. 

Clearly, every point is regular and of the sanie dimension in a neighborhood of 
a regular point. By the proposition, evcry singular point lies in the closure of the 
regular points. The following definition therefore makes sense: 
Definition 6. The dilnension of an almlytic set A C C , A  0, at ao is 
lira sup diln(A), 
aao 
where a tends to ao through regular points. çb define the dimension of A by 
dira A = sup dima A. 
aGA 
If d = diln A = infaeA dima A, we say that A is purely d-dimensional. 
Theorem 9. Suppose A  0 is analytic in the open set U C Ch and S(A)  O. 
Then 
dira S(A)  dira A - 1. 
An analytic set is called irreduczble if it calmot be writtel  a union of two 
analytic sets A = At UA2 with both A and A.2 proper subsets of A. Every analytic 
set can be decolnposed into irreducible colnponents: 
Theorem 10. Suppose A is analytic in U c C. Then A bas a unique representa- 
tion A =  A as a locally finite union of irreducible subsets, with A  A for 
a  ff. This decomposition is 9iven by 
{Aa} = {Ç  Ç is a connected coponent of A  S(A)}. 
Here the closure C is taken in U. 

Remark 1. It follows directly fronl Theormn 10 that if A is analytic in U C C and 
of dimension 0, then A is discrete in U. 
Most of the work to see that ,(ïl(c) is analytic in Q1 has been carried out 
in [DH2]; in fact, they prove (Corollary 2, page 313) that for any analytic family 
(f),e^ of quadratic-like mappings, and any parameter c in the Mandelbrot set. the 
set of parameters k for which fA is hybrid equivalent to z u + c is an analytic subset 
of A. The definition of an analytic family of polynomial-like maps is the following. 
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Definition 7. Let A denote a complex analytic manifold and suppose f = (fA " 
U --, U)E^ is a family of polynomial-like mappings. Set H' = {(A, z) I A E A, z 
U,}, L/= {(A,z) I A  A,z  U),} and f(A, z)= fA(z). We say that f isan analytic 
family of polynomial-like maps if the following three conditions are satisfied: 
(1) L/and L/' arc homeomorphic over A to A x D; 
(2) the projection 7r A from the closure of L/' in L/to A is proper; and 
(3) f  L/' -- L/is complex analytic md proper. 
For parameters a  Q, we bave that wl-renormalizations of fa locally form an 
analytic family. More precisely: 
Lemma 1. Given a0  Q, there are a bi-disk A C Q1 containing a0 and 
renormalizations (fa " Ua' -- Ua)aeA that form an analytic family of quadratic-like 
mappings. In particular, f-l(coe) is an analytic set in 
Of course a sinfilar statement about Q2 is true, and it follows that Q and Q2 
are open sets. 
Proof. Lct a0 E Q. Thon thcre are topological disks U, U' with toi(a0) E U' such 
that fao " U' -- U is quadratic-likc. Shrinking U and taking for U' the component 
of fao(U) that is compactly containcd in U, we may assume that the boundaries of 
U and U' are simple closed analytic curves and that the critical values fao(W(a0)), 
i = 1,2, are not contained in OU. Choose a bi-disk A around a0 such that for ail 
a in a neighborhood of A, the critical values avoid (OU. VVe then have a unique 
holomorphic motion h  A x (OU' - C of (OU' such that fa(h(a, z)) = fao(z)  (OU. 
Since the image of a compact set under a holomorphic motion depends continuously 
on the parameter (in the Hausdorff metric on compact subsets of the sphere), we 
may suppose that ha(OU') is contained in U for all a  A, by shrinking A if 
necessary. Note that h(a, OU) is a Jordan curve for each a  A. and denote by 
U a' the domain bounded by it. Then fa " Ua' -- U is a quadratic-like mapping for 
each a  A. Set L/' = {(a,z) I a E A,z  U'} and L/ = A x U. We claire that 
(fa " Ua' -- Ua).eA is ail a, mlytic fal, fily of quadratic-like mappings. To see this we 
must verify (1)-(3) in the definition. 
Let b " U  D be a homeomorphism. Then ff(a, z) = (a. b(z)) is a homeo- 
lnorphisln over A lnapping L/ onto A x D. Let b  D + U a' be the conformal 
map that takes 0 to w(a) and whose derivative b(0) is real and positive. Then 
 (a, z) = (a, ba(z)) is a homeomorhisn over A, mapping A x D onto L/'. This shows 
(). 
Note that the closure of L/' in L/is ' = (a,z) I a  A,z  U'a}. This follows 
since Ua depends continuously on a  A in the Hausdorff topology. Let A C A 
denote a compact set. Then the preimage of this set under the projection of/' 
onto A is {(a,z) I a E A.z  U'a}. Since U' a depends continuously on a, this set is 
compact. This shows (2). 
Çlearly f  (a, z) - fa(z) is complex analytic. Note that f(a, z) = fa(z) maps 
' \ L/' onto A x (OU. Hence the preimage ff' of a COlnpact set K C L/avoids/' \ 
Using that rrA(tç') = rc^(K), it follows that K' does not intersect the boundary 
of L/'. No K' is a closed and bounded subset of (I2 2. We conclude that f is proper, 
which finishes the proof of (3). 
That ,(ïl(coe) is analytic in 1 now follows directly by applying Corollary 2 on 
page 313 in [DH2] and Proposition 6. [] 
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FIGURE 2. To the upper left is drawu ma approximation of the 
Julia set of z H z 2 - 2. Below is the "rabbit", that is, the Ju- 
lia set of z H z 2 +c with c  -0.12236 +0.74486i. To the 
right is illustrated the Julia set of fa(z) -= z a - 3a2z + b, with 
a = (a,b) , (-0.03124478736- 0.8615522432i,-0.1514859674 + 
0.3334260962i). This cubic polynomial is the result of intertwining 
the two ment.ioned quadratic polynonfials at their/5-fixed points. 
Notice the copies of the Julia sets of the quadratic maps inside thc 
Julia set of the cubic polynomial. 

Next we are going to state a result from [EYI (see also [Ha]) tha shows tha Julia 
sets of quadratic polynonlials show up inside Julia sers of certaiu cubic polynomials. 
To do so, we first ueed some notation. From the Straightening Theorem we know 
that for a E Çi, an wi-renormalization has a unique fixed point corresponding to 
the /-fixed point. This point we denote by fli(a). 
The following result is an immediate consequence of the work of Epstein and 
Yampolsky in [EY]. They obtain it using intertwining surgery, although for this 
particular case the word "joilfing" surgery nfight be lnore appropriate since two 
quadratic-like mappings are joined together at their/3-fixed points; see Figure 2. 

Theorem 11. There is a map B: ItI\ {1/4} x ]I\ {1/4} -- Ç1Ç) Ç2, ?3dhCh i8 
a homeomorphism onto ifs image, mapping (c, c2) to a, where a satisfies fl(a) = 
fl2(a) and x(a) = ci, i = 1, 2. [] 

We can now finish the proof of Theorem 7. 
Let Q denote the component of Qa that contains B({c} x (_ai \ {1/4})). The 
set A = (Xl]Qi)-(c) is analytic in Ç. This set cannot be of dimension 2. If it 
were, then it would contain a bi-disk, which contradicts that B: (M\ {1/4})x 
{1/4})  Q' is a homeomorphism. Since A contains the homeomorphic image of 
M\ {1/4}, it follows that A has dimension 1. In particular (see Remark 1), the set 

of singular points S(A) is discrete in Ç. 
Let M = {c E Iii I k(Pc(O) = ff(c))} 
of P : z  z 2 + c. A standard normality 
that the boundary of Al is contained in the 
[CG]) shows that this set is dense in OM. 
and we deduce that infinitely many points 

, where ff(c) denotes the fl-fixed point 
argument (such as the one that shows 
closure of the tenter parameter, see e.g. 
Now, no point in Ad C M is isolated, 
in the image B({c},.M) are regular. 
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Taking a0 for a regular point in B({cc} >(J), the polynomial f has the required 
dynamical properties. This proves Theorem 7. 
Relating the dynamical plmle to a parameter slice. Let 1 --, C 2, t  a(t), 
denote a paralnetrization of the complex submanifold given by Theorem 7, with 
a(0) = a0. Abusing notation, we write ft = ftt), l(t) = l(a(t)) and w(t) = 
w(a(t)), i= 1,2. 
For all t  , we have that ft is l-renormalizable and the renormalization 
is hvbrid equivalent to P«. Ve dcnote by Kl(t) the filled Julia set of the Wl- 
renorlnalization of ft. This set lnoves holomorphically with t  
Theorem 12. There is a holomorphic motion h  Dx C  C such that ht(Kl(O)) = 
h'(t) and ht commutes witb the dgnamics on K(O), i.e., 
ht o fo = ft 
on K  (0). 
Proof. Choose a period p and denote by Zp(t) the set of points in K(t) that 
periodic of pcriod p under 
Lct çv = {(t,z)t  ,z  Zp(t)}, and let " ÇÇ  Ç denote the projections 
such that for cvery a  C  Ç we have that a = ((a),(a)). X claire that the 
projection   çp  , (t, z)  t, is a covering map. Let t0   denote an arbitrary 
point. Anv w-renorlnalization fro " U' 
is constant. The set Zp(to) contains no indifferent cycles, for the saine reason. It 
then follows from the hnplicit Function Theorem that there are a neighborhood 
W of t0 and holomorphic functions zi " Il"  C i = 1 .... , #Zv(to), such that 
z(t)  Zv(t ) for all i. Shrinking W, we can sulne that z - z is non-vanishing 
for i ¢ j. That #Zv(t ) is constant implies that i{(t,z) ] t   z  z(lV)} is an 
open neighborhood of the fiber [a(t0). The claire now follows because for each i 
the map ," {(t.z) ] t e II:z e zi(W)}  Il is a homeomorphism. 
Using that the projection r  çp  is a covering map, we can li   
 x Zp0)   to ri " X  and obtain a mapping H" 
Then define h   x Z(0)  C by h = r2 o H: i.e., such that the following 
diagram conmmtes: 
 z,0) 
The map h is a holomorphic motion: By the Implicit nction Theorem, t  h(t. z) 
is holomorphic for each z  Zp(0). Suppose hto(X) = hto(Y). Now t  (t. h(t. x)) is 
a lift ofthe identity to the map r " çp  , and the saine is true for t  (t, h(t, y)). 
So by uniqueness of liftings, h(t, x) = h(t, y) for all t. Since z  h(0, z) is injective, 
it follows that x = y. Thus z  h(t, z) is injective for all t 
The saine line of arguments shows that h commutes with the dynamics. Both 
the map t  (t, ft o h(t, x)) and t  (t, h(t, fo(x))) are li.s of the identity map to 
r  ç   that agree for t = 0, and by uniqueness they agree for all t. Thus the 
equation 
h(t, fo(x)) = ft oh(t,x) 
holds for ail t   and ail x  Z(0). 
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Noting that for each t the sets Zl(t), Z2(t),... are disjoint, we get a holomorphic 
motion h  D x Up Zp(O) that comnmtes with the dynamics. 
By the Straightening Theorem, Jp Zp(O) is dense in K 1 (0), so by the A-lemma 
(see [MSS]) we can extend h continuously to a hololnorphic lnotion h  D x h 1(0) -- 
C. Now b.(t, KI(0)) = Kl(t) by continuity; and, also by continuity, the extension 
colnnmtes with the dynanfics. 
Finally, using Stodkowski's Theoreln (see [S1]), we can extend h to h  D x C - 
C. [] 
Define the COlmectedness locus ç of the family {ft}te by 
S = {t e D I Iç(f) is COnlmcted}. 
Remark 2. Using a theorem of McMullen (see [Mc3]), it can be shown that 
contains a quasiconformal image of the connectedness locus llld C Ç. of the falnily 
{z  z d + c}«¢e, with the image of the boundary of llld contailed in the boundary 
of 8, for some d _> 2. 
We will hot use or prove this fact; instead we will show that the connectedness 
loc,ls also contains a copy of J. For this we need a map relating the paralneter 
disk {t  ID} to the dynamical plane {z  Ç} of fo- 
Lemma 2. Let ht(z) = h(t, z), and denote by k the preperiod of w2(O) under fo. 
Then the mapping H  ID -- C, t - h 1 ¢)fî(w2(t)), is quasiregular on domains 
compactly contained in ID and non-constant. In fact, H is t(-quasiregular on { t < 
p},p < , it I = ( + p)/O - p). 
Proof. Let us first show that H is hot constant. If it were, then for ail t  ID 
we would have fî(w2(t)) = h(/3(O)) = /3(t), where the la,st equality is due to the 
fact that h COlnmutes with the dynamics. However, by Theorem 11 we can find a 
sequence of parameters t, converging to 0, such that ft is w2-renormalizable and 
the critical point of the renormalization is hot prefixed. 
Let us now show that H is quasiregular on dolnains compactly contained in ID. 
To sec this, we will use the fact that the time t map of a holomorphic motion is 
quasiconformal. Indeed, it follows from Proposition 5 of [D] (compare with the 
£-lemma in [MSS]) that for [t] < p < 1 the mapping ht is K-quasiconformal with 
Iç = (1 + p)/(1 - p). 
Taking the o#--/distributional derivate of the equation 
bi ¢) H(t) = fî(w2(t)), 

we get 

0 0 OH 0 Of I 0  
-ht[H(t) ÷ zht[H(t)--(t) ÷ -zht[H(t) O -- -f; (w2(t)). 
Since t - h(t, z) and t  fî(w2(t)) are holomorphic functions, it follows that 
0 OH 0 
Rearranging and taking absolute values, we obtain 

Ott/o Oht/O2 

So H is K quasiregular on {Itl < p}, for p < 1, with K = (1 + p)/(1 - p). [] 
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Main theorem. We can now prove the main t.heorem: 
Main Theorem. There exist two cubic polynomials f, g which bave no indiffer- 
ent cycles and whzch are not topologically conjugate on their Julia sers, but which 
nevertheless bave the saine rational lamination. In particular, f and g are not 
combinatorially rigid. 
Pro@ We denote by b : U' -- V' a hybrid equivalence that maps a neighborhood 
U' of K(0) to a neighborhood V' of K(Pc) and conjugates fo : U' -+ U to Pc- 
Using that b is a conjugacy and Proposition 5, we get that/31 (0) is accumulated by 
a.rl)itrary small copies of J --- b-l(J), each of which is being mapped bijectively 
to . by an iterate of fo- Since H is quasiregular and not constant, it maps 0 to 
Pi(0) with a local degree d > 0. The copies of J are full and do hot contain 
and from this it follows that we can find a copy J' and an iuteger 1 > 0 such that 
 f0  maps .Ï' bijcctively onto J; md 
 H-l(joe) intersected with a neighborhood of 0 is the disjoint, union of d 
continua C't,..., Ça such that H maps each Ci bijectively onto J'. 
By bijectivity there is exactly one point ri Ç C1 such that PC ob(t) = 0 Ç J. 
Since h comnmtes with the dynamics on K  (0), we get 
By injectivity of HIc, we get that if t 2  C, and t2  t,, then 
We claire that f -- ff, and g -- ft for t2 Ç Ci \ {t, } will satisfy the requirements 
of the theorem. We must prove: 
(a) f and g do hot possess indifferent cycles; 
(b) f and g are hot topologica]]y conjugate on their Julia sers; and 
(c) f and g have the saine rational ]amination, AQ(f) = AQ(g). 
Part (a) is an immediate consequence of the following lemma: 
Lemma 3. Suppose t Ç D. If there exists an integer k > 0 such that ft(w2(t)) 
Içl(t), then 
(1) the filled Julia set of ff bas no interior: K(ft) = J(ft), and 
(2) ail cycles of ft are repelling. 
Part (b). Suppose there exists a homeomorphism r : J(f) - J(g) that conju- 
gates f to g on J(f). By Le,nma 3, L'(f) = J(f) and L'(g) = J(g). The critical 
points wi(tl) are distinguished in L'(f) in that they are the only points that have 
only one preimage. The critical points wi(t2) are likewise distinguished in L'(g). It 
follows, since r is a conjugacy, that .r maps the two critical points of f onto the two 
critical points of g. By equation (1), we nmst have either g}+i(w2(t2)) = wl(t2) or 
g+(w,(t2)) = w2(t2). However, we have alreadv seen that g+(w2(t2))  wl(t2), 
and we cannot have g}+i(wl(t2)) = w2(t2), since all the forward images of wl(t2) 
are contained in Kl(t2) and w2(t2) is hOt an element in Kl(t2). This proves (b). 
Part. (c). To complete the proof we shall show that the polynomials ff have 
the saine rational lamination for all t  C. We will show that every rational ray 
lands, and that the landing point depends holomorphically on t for values of t in a 
neighborhood of C. This will follow from the following stability result (Proposition 
2 of exposé 8 in the Orsay Notes, [DH1]). 
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Proposition 13. Let Po E 7)d and 0  Q/E. Assume that Roeo(O ) lands at the 
(pre)periodic point "/o  J(Po) and that the corresponding cycle is repelling. If 
P(7o) is hot a critical point for any i >_ 0, then there is a neighborhood A of Po in 
7) d such that 
(1) the ray R,(O) lands af a (pre)periodic repelling point 7(P) for all P  A; 
(2) the map A × { _> 0} -- C that maps (P, 1) to the point of potential  on 
the ray ROe(O) is continuous and holomorphic in P. [] 
Indeed, fix a rational anglc 0 and a parameter t0  C1. By Proposition 1, the 
dynamical ray of angle 0 lands at a (pre)periodic point that is either parabolic or 
repelling. We have seen that ft has no indiffercnt cycles. So to apply Proposition 
13 we just have to see that the (forward) critical orbits do not contain the landing 
point. This foliows easily, since thc forward orbits of the critical point w(to) are 
trapped in a forward invariant region on which ftl is conjugate to Pc : J -- J. 
By Theorem 4, J contains no (pre)periodic points. So the critical orbits do hot 
contain any (pre)periodic point such as thc landing t)oint of Rto (0). 
We conclude that the ray Rt(O) kceps landing and moves holomorphically in a 
neighborhood of each point in C. 
Suppose that the rays of angles 01 and 02 land at distinct points, for a parameter 
t0  II}. Because the landing points depend holomorphically on t in a neighborhood 
of t0, they will keep on landing at distinct points for all parameters in a neigh- 
borhood of t0. Conversely, suppose two rational rays land at the saine point for 
a parameter t0 E C. Since the rays are disjoint and the rays including landing 
points depend holomorphically on t in a neighborhood of t0, we get from Hurwitz's 
Theorem that the two rays keep landing at the saine point for all parameters in 
a neighborhood of t0. Noting that C is commcted, we conclude that if two rays 
land at the saine point for a parameter t0  C1, they will do so for all parameters 
t  Ci. So AQ(ft) = AQ(f0) for all t  C. [] 

We still have to prove Lemma 3. To do so we will use the fact that the multiplier 
at an indifferent fixed point is a quasiconformal invariant. 

Lemma 4. Suppose the two holomorphic germs gl,g2 " ((, 0) --- ((, 0) have in- 
different fixed points af the origin, and a quasiconformal germ conjugates gl to g2. 
Then the multiplier g (0) is equal to the multiplier g(O). [] 

The lemma is well known; in fact, Naïshul' [Nai] proves t.hat the nmltiplier of an 
indifferent fixed point is even a topological invariant. 
We now prove Lemma 3. 

Proof. It follows from the classification of Fatou components that (2) implies (1); 
so we need only prove (2). First note that K  (t) does hOt cotain any non-repelling 
cycles. Indeed, the wl-renormalization of ft is hybrid equivalent to Pc. So if 
K 1 (t) contains an indifferent cycle, then also Pc possesses such a cycle by Lemma 
4. But Theorem 4 asserts that all the cycles of Pc are repelling. 
We distinguish two cases, supposing ft has a non-repelling cycle. 
Case 1: The cycle is contained in J(ft). Then it is indifferent. Since both critical 
orbits are captured by Ex(t), in the sense that fï(w(t))  K(t) for all n _> 0 and 
fï(w(t)) E K(t) for ail n _> k, the cycle must be contained in Kl(t). We have 
seen that this cannot be the case. 



3638 CHRISTIAN HENRIKSEN 

Case 2: The cycle is contained in the interior of K(ft). Again, since the orbits 
of the critical points are capt.ured by K 1() and (by TheoreIn 4) this set has no 
interior, the cycle caimot be attracting. By the classification of Fatou coinponents, 
it is necessarily a Sicgel cycle. The boundary of the cycle of Siegel disks is contained 
in the closure of lhe postcritical set, and thus in K l(t). Since K  (t) is a full set, 
the Siegel disks heinselves are also contained in K  (t). As before, this is hot the 
case. [] 
Question. I pass on a question I was asked i)v McMullen during a presentation of 
the result of this pat)er. 
Question 1. Is there a natnral çombinatorial object associated t.o a polynomial 
f  Ca, d > 2, hat mfiquely deterlnines the dynamics of f on J(f)? 
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ZERO ENTROPY, NON-INTEGRABLE GEODESIC FLOWS 
AND A NON-COMMUTATIVE ROTATION VECTOR 

LEO T. BUTLER 

ABSTRACT. Let f] be a 2-step nilpotent Lie algebra; we say f] is non-integrable 
if, for a generic pair of points p, p E ri*, the isotropy algebras do hot commute: 
[Op, Op'] t O. Theorem: If G is a simply-connected 2-step nilpotent Lie group, 
9 ---- Lie(G) is non-integrable, D < G is a cocompact subgroup, and g is a 
left-invariant Riemannian metric, then the geodesic flow of g on T*(D\G) 
is neither Liouville nor non-commutatively integrable with C ° first integrals. 
The proof uses a generalization of the rotation vector pioneered by Benardcte 
and Mitchell. 

1. INTRODUCTION 
Let E be a C M manifold, let Z(E) dcnote the set of C M Rienlamlian metrics on E 
with zero topological entropy, and let I(E) denote those metrics with an integrable 
geodesic flow. Following Paternain [18], we can ask the questions: 
Question A: I.s the set of integrable geodesic flows I(E) nonempty? Is t]e set of 
zero-entropy geodesic flows Z(E) nonempty? 
Question B: What is the topological structure of I(E), resp. Z(E)? 
The recent example in [3] by Bolsinov and Taïnlanov showed that I(E) g Z(E). 
In this paper, it is shown that the reverse inclusion is false, too: Z(E) g I(E). 
Thus, zero topological entropy is neither a necessary nor a sufiïcient condition for 
integrability. 
There are well-known topological obstructions to the existence of a zero-entropy 
geodesic flow on a compact nlanifold: exponential word growth of the filndamen- 
tal group and exponential growth of the Betti nunlbers of the loop space (over 
any field) [18]. An interesting class of manifolds where neither of these two well- 
known obstructions is effective is the class of nilmanifolds: quotients of nilpotent 
Lie groups. The fundamental groups of these manifolds are of polynomial word 
growth [1], and they are aspherical; so the loop space has trivial homology in all 
but the 0-th group. This paper constructs 2-step nilnlanifolds on which ail left- 
invariant geodesic flows are non-integrable, but their entropy is zero. It is unclear 
if these manifolds adroit any geodesic flows that are integrable. It seems likely that 
they do hot. 
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There has been some work related to the theme of this paper. Eberlein, Lee 
and Park, and Mast (amongst others) have studied the question of whether a left- 
invariant Riemannian metric g on the two-st.ep nilpotent Lie group (7 bas a dense 
set of periodic points on S(F\G) for ail lattice subgroups F [10], [11], [15], [17]. 
One way to understand this work is that the authors use the integrability of the 
geodesic flows in question to prove their results; the use of the formalism of the 
Euler-Lagrange equations on TG rat.her than Hamilton's equations on T*G obscures 
this faet, however. In [4], [5], [6], explicit examples of integrable geodesi« flows are 
constructed on a number of families of 2-step nihnanifolds including those studied 
in [10], [11], [15]. In [6] it is also shown that every left-invariant geodesic flow 
on every 2-step nihnanifold bas zero topological entropy. The paper [7] constructs 
completely integrable, zero-entropy geodesic flows on an n-step filiform 1 nilmanifold 
for ail n, thus proving that neither the st.ep length nor the growth of the step 
length relative to the dimension of the group is an obstruction to the existence of 
integrable or zero-ent ropy geodesic flows. Finally, in [8], examples are constructed of 
n-step nilnmnifolds, n > 3, with left-invariant metrics that have positive topological 
entropy. 
The present paper constructs explicit examples of 2-step nilnmnifolds whose left- 
invariant geodesic flows are hot integrable. 
Let us fornmlate the results of this paper more precisely. Let  be a Lie algebra 
andp (5 *; denotebv gp = {z (5 9  adp = 0}. Thepoint p (5 * is re9ularif 
din  is minimal; a pair of points p,p'  * is 9eneric if dim[gp. 9'] is mininml. 
Definition 1.1. Let 9 be a 2-step nilpotent Lie algebra;  will be said to be non- 
inte9rable if, for a dense set of generic pairs of points p, p'  9*, we have [9», »']  0. 
Lennna 3.2 proves that Derinition 1.1 is equivalent to the existence of a generic 
pair of points p, p'  * such that [9, »'] ¢- 0. A notion related to non-integrability 
is alraost non-singularité:  is almost non-singular if there exists a regular point 
p  * such that the 2-form dp derined for ail ,y (5  by dp(z,y) = -(p,[x,y]} 
induces a symplectic form on /[, ] [15]. If  is ahnost non-singular, then for 
ail regular points p  *, » = Z(g). Non-integrable 2-step nilpot.ent Lie algebras 
cannot be almost non-singular 2-step nilpotent Lie algebras. There are, however, 2- 
step nilpotent Lie algebras that are neither ahnost non-singular nor non-integrable. 
The example of g = span {x, y,..., Yn, zl,---, Zn} with the relations [x. y] = z is 
studied in [41, [51. 
Here are two elementary constructions of a non-integrable 2-step nilpotent Lie 
algebra. Let [3 be a real, senti-simple Lie algebra and let 9 = [3[3 be the vector space 
direct sure of 2 copies of [3, with Lie bracket defined by [x @ y, x' @ y'] = 0 @ [x, x']. 
The example to keep in mind is [3 = so(3). A second example is this: let V be a 
real, odd-dimensional vector space and let A2(V) be the vector space of alternating 
forms on V* and let  = V @ A2(V) with Lie bracket Ix @ y, x' @ y'] = 0 @ x A x'. 
The lowest dimension for a non-integrable 2-step nilpotent Lie algebra is rive: let 
g6 - R3 ( A2(R3) and let Z C Z(g6) be a one-dimensional subalgebra; then 
g5 := g6/Z is also non-integrable. One can verify that there are no further examples 
in dimensions 3 or 4. 

1The step length of a nilpotent Lie algebra of dimension d >_ 2 is bounded by d- 1: it is filiform 
if it attains this bound. 
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Definition 1.2 (cf. [14]). Let et " _Iii --, _Iii be a C ° flow. It is integrable if there 
exlsts an open dense subset L C _Iii such that L is a C ° torus fibre bundle, and the 
fibres of L are Ct-invariant. 
The hypothesis that L is a C ° torus fibre buudle means that there is a C ° 
manifold B and L is a locally trivial fibre bundle over B with fibres - T . The most 
common way to obtain integrable systems is via the Liouville-Arnol'd-Nehorogev 
theorem for Hamiltonian systems. 
Theorem 1.3 (Main Theorem). Let G be a szmply-connected 2-step nilpotent Lie 
group, t : Lie(G) be non-integrable, D < G be a cocompact subgroup, and g be 
a left-invariant Riemannian metric. Let  : D\G. Then the geodesic flow of g 
on AI = T*E is not integrable in tle sense of Definition 1.2. In particular, it is 
neither Liouville nor non-commutatively integrable with C ° first integrals. 
Let us observe that hot all nilpotent Lie groups have discrete cocompact sub- 
groups; the existence of such a subgroup is known to be equivaleut to the existence 
of a Q-structure on t [16]. 
1.1. Outline. The proofof Proposition 1.3 is surprisingly straightforward and is re- 
lated to a generalization of a rotation vector to non-commutative groups. Schwartz- 
man [20] (sec also [19]) introduced the notion of an asymptotic homology class in 
Hl(_]ll;R) for any curve c" R ---, _Iii. Recall that HI(-]II;R) = rr/[rr, rr] ®R, 
where rr = rrl(/tl). This asymptotic homology class was generalized by Benardete 
and Mitchell [2] - who were building on the work of Chen [9] - to an asymp- 
totic homotopy class of the curve c. Let us sketch the definition of this homotopy 
class. This principally requires an explanation of where it lives. Assume that 
7r is a finitely-generated group. Let 50 := 7r and 5,+1 :: [rr, 5,]. In general, 
7r/Su has torsion, but the radical of 5,, A, : v/, is a nornml subgroup of rr 
such that rr" :: rr/A, is torsion-free. It is known that rr ' is finitely-generated, 
n-step nilpotent and torsion-free. By a theorem of Mal'cev [16], there is a con- 
nected, simply-commcted n-step nilpotent Lie group N, such that rr ' is a discrete, 
cocompact subgroup of N,» The group N, is unique up to isomorphism. Note 
that 71 "1  HI (]tl; Z)/Tor HI (]il; Z) and that NI - H1 (]il; R). This construction, 
called the Mal'cev completion of 7r, is a generalization of the standard tensor prod- 
uct. The asymptotic homotopy class defined by Benardete and Mitchell lives in 
the connected, simply-connected 2-step nilpotent Lie group N2. Let us note that 
the construction of the groups N, and A is "pointed'" because 7r = 7r(/l; q) is 
pointed: so we should use the notation {N,,q, A,,q}. A particularly nice feature 
of Benardete and Mitchell's paper [2] is that they embed t.he family of groups- 
with-lattice-subgroups {N,,q, A.n,q}qe M in a connected, simply-connected n-step 
nilpotent Lie group N, with lattice A and then show that there are isomorphisms 
Cq  N,.q ---, N, (which satisfy Cq(A,q) = A,) such that Cq, o ¢-1 is an inner 
automorphism of N, for all q, q'  ItI. Of course, there are no canonical choices for 
the 
The precise definition of the asymptotic homotopy class in N2 requires some 
work. There is no ergodic theorenl for these cocycles; so it is unknown if the 
asymptotic holnotopy class exists for almost all initial conditions. Because the 
interest is really in understanding the asymptotic holnotopy classes for a flow, we 
have elected to follow Pried [12] and define a related notion: projective homotopy 
classes. We show that it is possible to put enough algebraic structure on these 
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asymptotic homotopy classes in order to derive an effective necessary condition for 
integrability (see Lemma 2.7). Combined with a few calculations in section 3, this 
necessary condition gives a proof of Proposition 1.3. 

2. FREE PROJECTIVE ASYMPTOTIC HOMOTOPY CLASSES 
Let us continue with the notation in the outline. Let N be an n-step nilpotent 
Lie group and define FN to be the set of all one-parameter subgroups of N. Recall 
that if N is a connected, simply-connected nilpotent Lie group, then, by identifying 
N _ Lie(N) via the exponential map, we can identify FN with Fn = n/R, where 
n := Lie(N). Fn is the set of all subspaces of n with dimension at most 1. We 
can cquip Fn with the topology induced by the projection map 7r(x) = Rx, which 
makes it into a compact, commcted non-Hausdorff topological space. V'e will also 
denote the coset R.c by ,. The set {0} C Fn will be said to be trivial. Let us state 
the following: 
Lemma 2.1. Let L be a real Lie algebra, FL = L/R. Then 
1) if p : L  L is a linear transformation, there is a continuous rnap  : L  FL 
such that Tr o p =  o 
2) for ail :  FL there is a continuous rnap ad : FL  FL such that for ail 
.r  7r-1(): 7r  ad = ad  
The proof of Lemma 2.1 is straightforward. \Ve note that (1) implies that the 
adjoint representation of N on n descends to an action on Fn, while (2) shows that, 
if L = n is nilpotent, then we can introduce a grading on Fn by saying that ,  Fn a 
iff ade  = 0 and ad -k-1  . This grading is inherited from the grading of n by 
its derived subalgebras. Part (2) also implies that the projective bracket [, ] is 
well-defined; it is also well-defined to say that  and  commute. This is equivalent 
to Ix, y] = 0 for all x   and y  , which is equivalent to [exp(tx), exp(sy)] = 1 
for all s and t. 
We are interested in the case where N = Nn is the -th part of the Mal'cev 
completion of 7r = 71 1(]/; q). Let H : N -- Fn be defined by H = 7r  log. We want 
to define: 

Definition 2.2. Let c  [0, oc) -- N be a continuous curve, c(0) = 1. Let Ct := 
{H(«(s))  s k t} and define 
to be the projective homotopy class of c. If c'() := gc() for some g  A-, hen 
Noe hat  lies in any nonempty closed subset of n So Ici always contains 
the trivial subse ). For this reason le us define he following set mapping 
#(V) := (V)U {0} for any V ç n. 
w c «opt [«]  çoow: pt o o I.I on , a t 
x(t) = p(I og «(t)l - og «(t)) 
if c(t)  id and 0 otherwise. Then Ici = [d]. This shows that   2 E Ici iff there is 
a sequence xa E n of norm 1 such that d(ta) = xk and xa  x. Thus, the projective 
homotopy class is a generalization of the homology directions (a projective rotation 
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vector) of Fried. The following two lemmata might be interpreted to say that 
Definition 2.2 is "correct". 
Lemma 2.3. Let x E n, g  N, c(t) := e t, c'(t) := gc(t) and c"(t) :-- gc(t)g -1. 
Then Ici =/r(x) and [c'] = [c"] =/r(Adgx). 
Proof. The map II  N - Fn satisfies II(e t) = r(log(et)) = r(x) for t - 0. Thus 
Ct = {r(sx) " s >_ t} = r(x) and so Ct = r(x). The remaining claires are clear. [] 
Given a continuous nmp 7  T 1 - M we can extend 7 to a map R - M. Let 
this map be denoted by 
Let N be a connected, simply-connected nilpotent Lie group, D < N a discrete, 
cocompact subgroup and let E = D\N. For a = Dg there is a (non-canonical, non- 
unique) identification of r(E;a) with r(E; De) =- D by the map (t) - "(t)g -1. 
Lemma 2.4. Assume that D < N is a discrete, cocompact subgroup and let E = 
D\N. Let "  r(E;a) be a nontrivial homotopy class, a = Dg and let F  R -- N 
be a lift of  based at g. If' is homotopic to the loop t -o Dge tx, then IF] = 
#(Adgx). 
Proof. Assume that a = De, F(0) = e, and let d = F(1)  D. By hypothesis d - e. 
Then, there is a nonzero x  n such that d = e x. Let c(t) = etc; so that c(n) = d' 
for ail n  Z. The map q(t) = F(t)c(t) -1 is 1-periodic and continuous; so imq is a 
compact subset of N. Since r(t) = q(t)c(t), we can write that log F(t) = tx + p(t). 
The set imp is also compact; so log F(t)/I log r(t)l = x/Ixl + O(t -) is well-defined 
for ail t sufiïciently large. Then IF] =/r(x). 
In the general case, we non-canonically identify rl (E; Dg) with ri (E; De) via the 
map 7(t)  "(t)g -1 =: (t). From the arguments of the previous paragraph we can 
assume, without loss of generality, that 7(t) = Dge t. Then (t) = De tAdx. Apply 
Lemma 2.3 and the results of the previous paragraph to obtain the conclusion. [] 
The r de Rham theorem of Benardete and Mitchell allows one to introduce a 
second notion, which is more suitable for fiows. Let bt " M - M be a C 1 semi-fiow, 
and let q E M and n >_ 1 be fixed. The Mal'cev completion of ri (Mi q) induces a 
map fq,,  C°((R, 0), (M, q)) - N,. Let us denote a projective homotopy class in 
Fn by [-l,- We define the projective homotopy class of G(t) = ct(q) to be [fq,, oG] 
and denote this by [q],¢. In the sequel, we will drop the subscript n, ¢ when this is 
understood. Let us also introduce a convenient notation: if V C n (resp. W C Fn), 
then (_9(V):= {Adgv " g  N,v  V} (resp. (_9(W) = {Adgw : g  N,w  V}). 
It is clear that O(r(V)) = r(O(V)). 
Definition 2.5. The free projective asymptotic homotopy class of q is the set 
.T,,,(q) := ([q],,). 
By Theorem 3.1 of [2] and Lemma 2.1, if p = Cto(q), then [q] = Ad 9 [pi for some 
g E N,. Thus, the free projective asymptotic homotopy class is an invariant of 
the orbit {dPt(q)}tI:t. In addition, although the construction of Theorem 3.1 in [2] 
depends on the smoothness of the manifold M, the free projective homotopy classes 
are an invariant of the C ° conjugacy class of the semi-flow çbt. In certain cases, it is 
possible to see this directly: when I is a vector bundle over a nilmanifold, the free 
projective asymptotic homotopy classes can be defined directly from Definition 2.2 
without recourse to Theorem 3.1 of [2]. Note that Lemma 2.4 implies that Definition 
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2.5 is also "correct": if et(q) -- qe tx, then the projective asymptotic homotopy class 
of Cq(t) = qe ix is [Cq] = /r(Adqx), while the free projective asymptotic homotopy 
class is 9fo(q) = ([C]) = UgeN [C]. 
Let us now explore the relationship of 9rn,¢(q) with integrability. 

Definition 2.6. Let U, V C l?n. We say that U, l/ weakly commute if there exist 
nontrivial  E U, f  V such that [, ] = . 

Lemma 2.7. Let bi be a compact manifold and et " bi --, 1I be a C ° semi-flow. 
Assume that there is a path-connected Ct-invariant set U C lil such that 7r(U) is 
abelian. Then for all u,   U and all n >_ 1, either 9rn,¢(u) and 9r,¢(u ) weakly 
commute or at least one of ',¢(u), 9r,¢(u ) is trivial. 

Pro@ Let i  U -* ]il denote the inclusion nap; i maps ;ql(U; u) into an abelian 
subgroup of tri (/il; i(u)) for all u  U. By the naturality of the Mal'cev completion 
of tri (M), for each n _> 1, i.Tr (U; u) gets mapped to a discrete torsion-free abelian 
subgroup A of A, and so there is an abelian subgroup A C Nn such that 
A = exp(span log(A)). The positive-invariance of U implies that 9r,¢(u) and 
'n,¢('u ) are subsets of A,» Therefore, if there arc nontrivial elements in both 
9r,,¢(u) and 'n,¢(u), theu there are nontrivial elements in each set that commute; 
so the sers weakly COlmnute. Otherwise, at least one of the sers is trivial. [] 

Corollary 2.8. Let et be a C ° semi-flow. Assume that there is a C ° embedding i of 
U - T k × D t into III, such that i(U) is Ct-invariant. Assume that there is a C ° map 
a  D t --, R  such that for all 0  T , I  D t we bave i-¢ti(O, I) = (0 + tw(I), I). 
Then for all u, u   U and all r >_ 1, either the sets ',,¢(i(u)) and ',,¢(i(u)) 
weakly commute or at least one is trivial. 

3. TWO-STEP NILPOTENT LIE GROUPS 

Let 9 be a 2-step nilpotent Lie algebra with center 3 = Z(9), so that [9.9] C Z(9), 
let (,) be an inner product on 9, and let 

be an (,)-orthogonal decomposition of 9- The Lie bracket on 1 is written as 
Ix + y,x+ y] = [x, xq for all x,x   [1 and y,y  , and so the colnmutator 
defines a skew-symmetric, bilinear form o., : [1 x b --,  by o.,(x, x') ---- Ix, x]. 
The Lie algebra 9 can also be given the structure of a Lie group ((7, .) by 
[X,Y], so that 9 Lie(G) and the exponential map is 
X*Y := X+Y+i = 
the identity. In the sequel, elements in G will often be viewed as elements in 
1 under the inverse (logarithm) map - which is the identity map in these coor- 
dinates. If D is a discrete, cocompact subgroup of G, then there exists a gen- 
erating set X,...,X,Y,...,Yq where I,...,Yq generate Z(D)and the cosets 
X + Z(D),...,Xp + Z(D) generate D/Z(D) and p = dimD, q = dim3 [16]. The 
generating set therefore determines a basis of 1 and an inner product (,) relative 
to which it is an orthonormal basis. It may be supposed then that (,) = (,)', 
[} = span R{XI, ... , Xp} aIld  = span R{Y, ..., t}. 

Lemma 3.1. Let D <_ G be a discrete, cocompact subgroup and let (,) be an inner 
product on . Then there exists an automorphism f " G --, G and a subgroup 
D' = f-(D) with generators X .... ,Xp, Y,...,Yq such that (X,Y.) = O. In 
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addition, if gis the left-invariant metric on G determined by (,), then ( D'\G, f'g) 
is isometric fo (D\G, g). 
Pro@ Let 9 = [7 @ 3 be the (,)-orthogonal decomposition of 9- Let a(x) be the 
(,)-orthogonal projection of x C h onto 3- The map F : x + y --, x - a(x) + y for all 
x C b and y  3 is an automorphism of 9. Let f = exp oF o log be the map induced 
by F on G. Then f is an automorphism and by construction F(h) is (,)-orthogonal 
to 3- [] 
Lemma 3.1 is proven in [13] for Heisenberg groups. The importance of this 
lemma is that, by fixing a discrete, cocolupact subgroup D with a fixed generating 
set, attention can be confined to those metrics that are block diagonal relative to 
this fixed basis of 
relative to which 9 = b @ à, D will be a discrete, cocompact subgroup of G with 
(, }-orthonormal generating set X .... , Xp,I,..., Y and (,) will be a second ilmer 
product that is block diagonal: for all X. X' 
(3.1) (X + Y,X 
where alij = (Xi,Xj) aud Bat = ()),). The metric g on G will be the left- 
invariant metric determined by (,) or, equivalently, the pair al,/3. 
Finally, let 9 be a reM, nonabelian Lie algebra and let r = minpe, dira 9p; r is 
comlnonly called the index of 9- It is well-known that the set of p where dira 9p > r 
is a nontrivial algebraic set; let 9,*- denote the complement of this set. Let G denote 
the Grassmannian of r-dimensional subspaces of 9, and let G = G × 9 denote 
the trivial G bundle over 9- The map p - 9p restricted to 9 is a section of G. 
Since x G 9p iff adp = 0 1 «,t= c,iP«X -- 0 for i = 1, n (where c, are the 
structure constants of 9) iff x satisfies a system of linear equations with coefficients 
linear functions of p, so it follows that the section p - 9p is algebraic in p. Thus, 
the map f" 9  9 defined by p - [gp, 9pi is algebraic and so the subset (f ¢ 0} 
is a Zariski-open subset of 9.. Consequcntly, (f ¢ 0} is either elnpty or else it is 
dense in 9- We have therefore proven: 
Lemma 3.2. Let 9 be a 2-step nilpotent Lie al9ebra. Then 9 is noninte9rable iff 
there exists two 9eneric points p,p'  9 such that dira [gp, 9p'] > 0. 
3.1. Geodesic equations of motion. Let 9 be a 2-step nilpotent Lie algebra. Let 
A: 3"  so(D) be defined for all x,x' C h and q  3" by (x,A(q)x') := (q,[x,x']). 
Observe that 
Lemma 3.3. ker A(q) = 
* I X I 
Pro@ x  kerA(q) C b iff0 = (q,[x,x']) = -(adq,x) for all  b. Since 
* t X/ 
9 = h @ Z(9), this is true iff 0 = (q, [x,x']) = -(adq, x } for all c 9- [] 
Lemma 3.4. 9 is nonintegrable iff there exists generic points q, q'  3" such that 
[ker A(q), ker A(q')] ¢ 0. 
Pro@ Apply Lemmas 3.2 and 3.3. [] 
Let (x,y,p,q) be the coordinates of a point in T*G = b × 3 × [7* × 3* The 
Hamiltonian of the metric g on T*G is Hg = ½(P, Rp) + 
and S =/3-1. The equations of motion are 
{ 1 
 = 0,  = s + [,p], 
(3.2) XHg= D =-A(q)Rp,  = Rp. 
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Then q is a 3*-valued first integral of X/g and F := p + A(q)x is an h*-valued first 
integral. One way to thiuk about the integrability of X/g is to determine in which 
cases F can be "pushed down" to T* (D\G). 
Let r 2 = R be a positive-definite, synmmtric square root of R (recall that 
is identified with h via the inner product (,)). Let B(q) = rA(q)r E so(b) and 
p = rv, which implies  = -B(q)v. Because B(q)  so([j), both ker B(q) and its 
(,)-orthogonal complement are B(q) invariant. Let Kq = ker B(q) and Lq = h'q -L. 
Then for each q  ,C, h = Kq @ Lq. Note that tçq = r - ker A(q). 
Let k = infqe, dimKq and l = 811pqt. dimLq, let Gs(h*) denote the Grass- 
man,6an of s-dimensional planes in h* and lct Gs = G,(h*) x 3" denote the trivial 
bundle. The following is obvious: there is a subset 3- C 3" such that 3" - 3 is a 
closed algebraic sct aud the map q -- tçq (resp. q - Lq) is an algebraic section of 
G * * 
],% (resp. G, la). 
For each v  * write v = Vo + v where Vo = vo(q) c= Kq and ri = v(q) c= Lq. 
Abuse notation, aud let B(q) - = (B(q)]Lq) - when h'  *. We tan integrate 
the equations 3.2 to obtain 
(a.a) 
{ 
q(t) = q, «(t)=y+tSq+-fg[x(s),rv(s)]ds, 
dpt(x,y,v,q) = v(t) : e-tB(q)v, x(t) = x + trvo + rB(q)-{1--e-B(q)}v. 
Note that x(t) = trvo+O(1) = tu+O(1), where u = rvo  ker A(q). The expression 
for y(t) may be expanded to yield: 
(3.4) 
l for 
Æ(t)=+tsq+- {[,rol+[,re-()%]} ris 
+ {s[rvo, re-sB(q)vl] + [rB(q)-l(1 _ e-sB(q))vl,rvo qt_ re-sB(q)vl]} ds. 
Inspection of the integrands shows that the first, second and fourth integrands are 
bounded functions of s. So their integrals bave a norm bounded by const, x t. The 
substitution u = srvo and dr = re-B(q)*vldS combined with integration by parts 
on the third integrand shows that its norm is bounded by const, x t, also. 
Lemma 3.5. Let P = (x,y,p,q)  T*G be such that B(q)  O. Let p = rv and 
wmte v = Vo + v (resp. p = Po + Pi) with Pi = rvi. Assume that Vo  O. Then the 
projective asymptotic homotopy class of dpt(P) is nontrivial and [P]2, C '(P0 + 
Proof. Let p : T*G -- G denote the canonical projection. Assume that (x, y) = 
(0, 0). Since lY(t)l <- const, x t, vo  0, x(t) = trvo +O(1) and Po = rvo, w have that 
PdPt(P)/IP¢(P)I = c(t)po+ z(t), were c(t) is a positive function of t bounded awy 
from 0 and oc and z(t)  5 has uorm < 1. Thus rr(p@t(P)) = rr(p0 + c(t)-z(t)). 
Since Ic(t)-z(t)l <_ const., this show that [P]2, Ç/(p0 + 
Assume now that g = (x, y) -J: (0, 0). The left-invariance of  and the results of 
the previous paragraph show that [P]2, C_ #(Adgp0 +5). Since tt is 2-step nilpotent, 
one computes that AdgPo + 5 = Po + 3, which proves the lemma. [] 
Corollary 3.6. Assume the hypotheses of the previous lemma. Then the free pro- 
jective asymptotic homotopy class of dpt(P) is nontrivial and )r2,e(P) c_ #(Po + 3)- 
Proof. Apply the previous lemlna and the definition of -. [] 
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Proposition 3.7. Assume that 9 is a nonintegrable 2-step nilpotent Lie algebra 
with a Q-structure. Let D < G be a discrete cocompact subgroup, E = D\G and 
g be a left-invariant metric on G. Then the geodesic flow of g on T*E is hot 
integrable. 
Proof. Let çt (resp. çbt) denote the geodesi« flow of g on T*E (resp. T'G). The 
geodesic flows satisfy t(DP) = DCt(P) for all P E T'G. Let DP, DP'  T'E. 
From the definition of 9 , .T',[(DP) = 9n,¢(P) and similarly for DP'. By Corollary 
3.6, {0} : 92,¢(P) Ç #(po(q)+ ,)and {0} : 92,(P ') Ç i(p'o(q' ) + 3)- Let 
us remark that if 2  92,(P) and 2'  92,(P ') are both nontrivial elenlents, 
then [2,2'] = u(o(q),po(q')]). By Lemnla 3.4 and the subscquent discussion, 
Po(q)  9q ç b and P'o(q')  9q' ç b. By the hypothesis that 9 is nonintegrable and 
2-step nilpotent, it follows that for an open dense set of (q, q') E a* x a* there exists 
a0  9q ç ri, u)  9q' ç b su«h that [a0, u)] : 0. Thus, for an open and dense set of 
(DP, DP')  T*E x T*E the free projective asynlptotic hoHlotopy classes 92,5(DP) 
and 92,(DP') are nontrivial an(l do hot weakly «Onlmute. 
Assume t : T*E -- T*E is integrable in thc sense of Definition 1.2. Then 
there exists an open dcnse sct L C T*E sueh that L = [.J La and the open sets 
La are çt-invariant and have an abclian fundmlmntal group. Thus, for an opcn 
dense set of points DP  T*E there is an open neighbourhood of DP, La(op) C 
L, which is ç)t-invariant and has an abeliatl fundanlental group. Since the free 
projective asymptotic homotopy classes of an open dense set of points (DP, DP')  
La x La are nontrivial, Lenlnm 2.7 inlplies that 2,(DP) and .T2.ê(DP' ) weakly 
conmmte. Therefore, there is an open set of points (DP, DP')  T*E × T*E with 
weakly commuting free projective asymptotic homotopy classes. This contradicts 
the conclusion of the previous paragraph, which is absurd. [] 

EFERENCES 

1. H. Bass, The degree of polynomial growth of Jïnztely generated nilpotent groups, Proc. London 
Math. Soc. (3) 25, 603-614 (1972). MR 52:577 
2. D. Benardete and J. Mitchell, Asymptotic homotopy cycles for jïows and Iii de Rham theory, 
Trans. Amer. Math. Soc. 338(2), 495-535 (1993). MR 93j:58107 
3. A. V. Bolsinov and I. A. Taimanov, Integrable geodesic jïows unth positive topological entropy, 
Invent. Math. 140(3), 639-650 (2000). MR 2001b:37081 
4. L. T. Butler A new class of homogeneous manzfolds unth Liouville-integrable geodesic jïows, 
C. R. Math. Acad. Sci. Soc. R. Can. 21(4), 127-131 (1999). MR 2001i:53141 
5. L. T. Butler New examples of integrable geodesic jïows, Asian J. Math. 4(3), 515-526 (2000). 
MR 2001i:37090 
6. L. T. Butler, Integrable geodesic jïows with wild Jïrst integrals: The case of two-step nilman- 
ifolds, Ergodic Theory Dynam. Systems, to appear. 
7. L.T. But|er Integrable geodesicjïows on n-step nilmanifolds, J. Geom. Phys. 36(3-4), 315-323 
(2000). MR 2002j:37077 
8. L. T. Butler Invariant metrics on a nilmanifold with positive topological entropy, submitted 
to Geometriae Dedicata. 2001. 
9. K. T. Chen. Extension of C °¢ function algebra by integrals and Malcev completion of 7rl. 
Advances in Math. 23(2), 181-210 (1977). MR 56:16664 
10. P. Eber|ein Geometry of 2-step nilpotent groups with a left invariant metric, Ann. Sci. École 
Notre. Sup. 27(5), 611-660 (1994). MR 95m:53059 
11. P. Eberlein. Geometry of 2-step nilpotent groups with a left invariant metric. II, Trans. Amer. 
Math. Soc. 343(2), 805-828 (1994). MR 95b:53061 
12. F. Fried, The geometry of cross sections to jïows, Topology 21(4), 353-371 (1982) MR 
84d:58068 



3650 L.T. BUTLER 

13. C. S. Gordon and E. N. Wilson, The spectrum of the Laplacian on Riemannian Heisenberg 
manifolds, Michigan Math. J. 33(2), 253-271 (1986). MR 87k:58275 
14. A. B. Katok, Ergodic perturbatzons of degenerate integrable Hamiltonian systems, Izv. Akad. 
Nauk SSSR Ser. Mat. 37, 539-576 (1973). MR 48:9758 
15. K. B. Lee and K. Park, Smoothly closed geodesics in 2-step nilmanifoids, Indiana Univ. Math. 
J. 45(1), 1-14 (1996). MR 97h:53044 
16. A. I. Malcev, On a class of homogeneous spaces, Amer. Math. Soc. Translation, no. 39, 1951. 
MR 12:589e 
17. M. Mast, Ciosed geodesics in 2-step niimanifoids, Indiana Univ. Math. J. 43, 885-911 (1994). 
MR 96a:53057 
18. G. P. Paternain, Geodesic flows, Progress in Math., vol. 180, Birkhuser, Boston, MA, 1999. 
MR 2000h:53108 
19. F. Rhodes, Asymptotic cycles for continuous curves on geodesic spaces, J. London Math. Soc. 
(2), 6, 247-255 (1973). Ml 49:6208 
20. S. Schwartzman, Asymptotic cycles, Ann. of Math. (2) 66, 270-284 (1957). MR 19:568i 

DEPARTMENT OF I[ATHEMATICS, NORTHWESTERN 
EVANSTON, ILLINOIS 60208 
E-mail address: lbutler@math.northwestern.edu 

UNIVERSITY, 2033 SHERIDAN ROAD. 



TRANSACTIONS OF THE 
AMERICAN MATHEMATICAL SOCIETY 
Volume 355, Number 9, Pages 3651-3667 
S 0002-9947(03)03280-X 
Article electronically published on Match 17, 2003 

COMPLETE HOMOGENEOUS VARIETIES: 
STRUCTURE AND CLASSIFICATION 

CARLOS SANCHO DE SALAS 

ABSTRACT. Homogeneous varieties are those whose group of automorphisms 
acts transitively on them. In this paper we prove that any complete homo- 
geneous variety splits in a unique way as a product of an abelian variety and 
a parabolic variety. This is obtained by proving a rigidity theorem for the 
parabolic subgroups of a linear group. Finally, using the results of Wenzel 
on the classification of parabolic subgroups of a linear group and the results 
of Demazure on the automorphisms of a tïag variety, we obtain the classifi- 
cation of the parabolic varieties (in characteristic different from 2, 3). This, 
together with the moduli of abelian varieties, concludes the classification of 
the complete homogeneous varieties. 

0. INTRODUCTION 

Let X be a variety over an algebraically closed field k. Let G be the functor of 
automorphisms of X that acts naturally on X. We say that X is homogeneous if 
G acts transitively on X, that is, if for each pair of points x, x' E Hom(S, X), there 
exists an automorphism - E Auts(X x S) (after a faithfully fiat base change on S) 
transforming one into the other: -(x) = x'. 
It is known that the group of autolnorphisms of a complete variety exists, i.e., 
the functor G is representable, and it is locally of finite type (see [7]). Hence, if 
X is smooth, connected and homogeneous, then the reduced connected component 
through the origin of G acts transitively on X. Therefore, every smooth and con- 
nected homogeneous variety is isomorphic to G/P, with G a smooth and connected 
algebraic group and P C G a subgroup. 
Let Aut°(x) denote the reduced connected component through the origin of the 
automorphism scheme of X. 
The main results that we obtain here are: 
(1) A complete homogeneous variety splits canonically and uniquely as a direct 
product of an abelian variety and a parabolic variety. 
A parabolic variety means a variety of the form G/P with G an aflïne, smooth 
and connected algebraic group and P a parabolic subgroup (eventually not 
duced), i.e., a subgroup containing a Borel subgroup of G. This is Theorem 5.2. 
The analogue of this result for compact Ktihler manifolds is due to A. Borel and R. 
Remmer (see [2]). 
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For the proof, it is necessary to prove that any action of an affine group on a 
variety with pojective olbits is trivial. More precisely, it is proved that if an affine 
algebraic group acts on a variety and the orbits are projective and equidimensional, 
then the variety splits as the direct product of an orbit and the quotient by the 
action. This is Theorem 3.3. Borel proved an analogue of this result for a complex 
variety (sec [1]). 
(2) From this result the next one, of great interest for the classification, fol- 
lows easily: The automorphism group of a homogeneous cornplete variety classifies 
(modulo isomorphi.sms) the variety, vp fo the choice, in such a group, of a class 
(vmdulo automorphism.s of the group) of a parabolic subgroup [P]. Moreover. thzs 
group of automorphisms splits, in a unique way, as the direct product of a semis[m- 
pie grovp of adjoint type (that is. with trivial ceter) ad an abelian variety, and 
the subgroup P is a parabolic s'ubgroup of the semis[tapie part. That is, to each 
holnogeneous variety X is assigned its autolnorphism group Aut°(X) = G x A and 
the class [P], modulo automorphisms, of a parabolic subgroup P C G; thus the 
maps X  (G,A, [PI) and (G,A, [PI)  X = G/P x A establish an equivalence of 
objects modulo isomorphisms. This is Thcorem 5.7. 
It should be noted that thc triplcts (G, A. [PI) are chosen with the single condi- 
tiou that Aut°(G/P) = G. Thcrcfo'e, a first qucstion is to know whether G may be 
any semisimplc group of adjoint typc; in othcr words, if G is given, does there ex[st 
a parabolic subgroup P C G such that Aut°(G/P) = G? The answer is affirmative, 
and is due to Demazure ([4]). From his results one obtains in particular: 
If G is a semis[tapie group of adjoint type and I3 is the variety of its Borel 
subgroups, then Aut°(13) --- G. 
In conclusion, the classification of homogeneous complete var[et[es is equivalent 
to the classification of abelian var[et[es, semis[tapie groups of adjoint type (both of 
them well studied, sec [8], [9], [6]) and parabolic subgroups P of a given semis[tapie 
group G (modulo automorphisms) such that Aut°(G/P) = G. Therefore, the 
classification is essentially reduced to the classification of (non-exceptional, in the 
sense of Demazure [4]) parabolic subgroups of a simple group of adjoint type. 
Finally, from the classification of the parabolic subgroups (in characteristic p ¢ 
2, 3) due to Wenzel ([12]) and the determination of the non-exceptional and reduced 
parabolic subgroups duc to Demazure ([4]), we shall classif.v the parabolic var[et[es 
in characteristic p ¢ 2, 3. 

1. KNOWN RESULTS 
The following results are well known and moEv be round in [5], [6], [8], [9], [11]. 
Theorem 1.1 (Barsotti-Chevalley). Let G be a smooth and connected algebraic 
group. There exists a unique normal subgroup G C G that is affine, smooth and 
connected, such that the quotient G/G is an abelian variety. 
Definition 1.2. An affine algebraic group is said to be linearly reductive if any 
linear representation splits as a direct sure of simple representations. 
An example of a linearly reductive group is the torus, and this is the onlv smooth 
and connected one in positive characteristic. 
Theorem 1.3. If a linearly reductive group acts on a smooth variety X, then the 
closed subscheme X  of the fixed points is a smooth subscheme whose tangent space 
at each point x is the subspace of the G-invariant tangent vectors of X at x. 
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Definition 1.4. A parabolic subgroup of an affine, smooth and connected group 
G is a subgroup P C G such that the quotient variety G/P is complete. 
Definition 1.5. A parabolic variety is a complete variety of the form G/P, where 
G is an affine, smooth and connected group and P C G is a subgroup. A parabolic 
variety is called a flag variety if P is a reduced scheme. 
Theorem 1.6 (Chevalley). If G is an affine group and H C G a subgroup, then 
G/H is a quasiprojective scheme and the action of (7 on G/H is projective, i.e., 
there exists a linear representation E of G sucl that G/H is a sub-G-scheme of 
(E). 
Remark 1.7. It follows froln this theorem that ally parabolic varicty is projective. 
Theorem 1.8. Any flag variety is a Fano variety; that is, the dual of the dualizing 
sheaf is ample. 
Theorem 1.9. Let G be a semisimple group, Aut (7 the automorphisrn group of G, 
and I C Aut G the subgroup of the inner automorphisrns of G. Then (Aut G)/I is 
finite. In particular, if G is of adjoint type, then the connected component tlrough 
the origin of Aut G is isomorphic to G. 
Theorem 1.10. Let X be an abelian variety, and Aut°X C Aut X the connected 
component tlrough the origin of the automorphism group of X (as a variety). Tle 
inclusion X "- Aut o X, x H tx = translation by x, is an isomorplism. 
Theorem 1.11 (Borcl's fixed point theorem). If a smooth, solvable and connected 
affine group acts on a complete var,ety, then the subscheme of the ri:ced points is 
hot empty. 
Definition 1.12. Wc shall dcnotc by Pic (X) thc scheme in groups (if it cxists) that 
paramctrizes thc invcrtiblc shcaves on X and by Pic ° (X) thc connectcd componcnt 
of Pic (X) through thc origin. 
Theorem 1.13. If X is a complete variety, then Pic(X) exists; it is a proper 
scheme and the tangent space at the origin is isomorphic to H a ( X, (_9x). Moreover, 
(1) If X is an abelian variety, then Pic(X) is smooth; lence, Pic°(X) is also 
an abelian variety. 
(2) If X is a parabolic variety, then Pic°(X) = O: in particular. Pic (X) is 
a discrete k-scheme (i.e., I_[Speck). Furthermore, if X = G/P where 
G is semisimple and simply connected, then Pic (X) is isomorphic to the 
character group of P. 
Let X be a variety such that Pic (X) exists. Let P be a universal invertible sheaf 
on X × Pic°(X), i.e., an invertible sheaf such that the pair (Pic°(X), 7 )) represents 
the functor of invertible sheaves of degree zero Oll X. By the universal property, 
7) is univocally determined up to inverse images of invertible sheaves (of degree 
O) on Pic°(X) by the projection X × Pic°(X)  Pic°(X). Hence, if one fixes a 
point x0 E X and requires that 7) be trivial on x0 × Pic°(X), then 7) is completely 
determined. 
Given 7), one defines qo" X --, Pic°(Pic°(X)) by qo(x) = 7).×p,O(x)" If, iii order 
to dctermine 7), one replaces thc choscn point x0 by anothcr one, x), thon one 
obtains the saine morphism composcd with thc translation on Pic°(Pic°(X)) by 
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the invertible sheaf ç(x)-1. That is, ç is univocally determined, up to translations 
on Pic°(Pic°(X)), by the choice of a point of X. If X is a group, one chooses as x0 
the origin of the group; this is equivalent to saying that ç is a morphism of groups 
(ç(0) =0). 
Theorem 1.14 (DuMity of abelian varieties). If A is an abelian variety, then the 
natural morphism A  Pic°(Pic°(A)) is an isomorphism. 
Theorem 1.15. If X, Y are complete varieties, then 
Pic°(X) × Pic°O ") = Pic°(X × 
and the correspondence is (£, £') - £ ¢-, £'. 
k 
2. LOCAL TRIVIALITY OF DEFORMATIONS OF PARABOLIC SUBGROUPS 
Let us assume that k is an algebraically closed field of arbitrary characteristic 
and let G be aa aflïne, smooth and comwcted algebraic group over k. 
Definition 2.1. Thc nilpotence degrce of an algebraic group G (over an alge- 
braically closcd field) is the ordcr of the quotient by its reduced subgroup: 
That is, it is the dimension of thc finite vector space of the fuuctions of 
It is known that a subgroup P C G is parabolic if and only if it coutains a Borel 
subgroup B of G. Moreover, given a Borel subgroup B, each parabolic subgroup 
has a conjugated one containing B, because all Borel subgroups are conjugate. 
Therefore, the problem is to determine the structure of the parabolic subgroups 
containing a given Borel subgroup. 
The aire of this section is to prove the local triviality of any deformation of 
parabolic subgroups (containing a Borel subgroup B). More precisely, let us denote 
Xs = X × S for each k-scheme S. Then, 
Theorem 2.2. Let S be a connected scheme and P C G8 a subscheme .in groups 
over S containing Bs. Then: 
(1) /f  is fiat over S, then if is constant; tbat is, ï = Ps for some subgroup 
P C G containing B. 
(2) If S is integral and the fibres of ï -- S bave constant dimension and 
nilpotence degree, then P is constant: P = Pz. 
One first observes that any parabolic subgroup contains the radical R(G) of 
the group (the maximal solvable, normal, smooth and connected subgroup of G). 
Hence, the parabolic subgroups are in biunivocal correspondence with the parabolic 
subgroups of the quotient G/R(G), which is a semisimple group. Therefore, we shall 
assume in the following that G is semisimple. 
Let T  B be a maximal torus of G, T the roots system of G associated with 
T, T + the positive roots (i.e., the roots of (B, T)) and $  T the basis of simple 
roots conta]ned in T +. 
Remark 2.3. The reduced parabolic subgroups (containing B) are in biunivocal 
correspondence with the subsets I of the basis $. In particular, the number of 
them is finite (exactly 2 Isl). Therefore, the theorem is almost immediate in charac- 
ter]stic 0, since any group is reduced, and hence the scheme of parabolic subgroups 
contain]ng B is finite and discrete. Thus, for the proof of the theorem we sball 
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assume that the characteristic is positive, p > 0, although the saine proof is valid 
in characteristic 0 with the corresponding simplifications. 
Let B be the variety of Borel subgroups of G. It is known that B = G/B, as G- 
varieties. Let U c B be the unipotent part of B and U- the unipotent part of the 
Bord subgroup B- opposite to B (that is, the unique Bord subgroup containing 
T and such that B ç B- ----= T). One has that U- is identified, as a U--scheme, 
with an open subset L/- of B (preciscly, with the open set of the Borel subgroups 
B' such that B' fb U- = {e}), which coincides with the unique open orbit under 
the action of B-. 
Lemma 2.4. Let S--, Speck be a base change. The map P H PçU, between the 
subschemes in 9roups of Gs over S containin 9 Bs and the subschemes in 9roups of 
U over S stable by T, is injective. Moreover, ftat subschemes correspond fo ftat 
subschemes. 
Proo]. Each subscheme in groups of Gs, ï D Bs, defines a closed subschcme 
P/Bs C (G/B)s = Bs, which deterlnilms P, since P is the preimage of that closed 
subscheme by the morphism to the quotient n: Gs --, Gs/Bs = Bs. Moreover, 
lg N (Ï/Bs) is dense in /Bs (be«anse it is dense in the fibres over S, since any 
parabolic subgroup is irreducible). Hence, P/Bs is the closure of L/ CI (P/Bs) 
in Bs. It follows that the closed subscheme 
However, the identification of U- with L/- induces another one of 
and it is clear that L/ ç (P/Bs) is identified with U ç P, which is a subscheme 
in groups of Uç and is stable mider the action of T by conjugation (since both 
subschemes are stable). In conclusion, U CI P determines P. The flatness of 
P CI U when P is fiat follows from the flatness of P/Bs C Bs (this one is fiat 
by descent, since ' --, Ï:'/Bs is faithfully fiat), since ' ç U is identified with the 
open subscheme ( P / B s ) ç glU. [] 
In the case that S = Spec k, this lemma may be obtained from Proposition 4 of 
[12] as a particular case. Therefore, this lemma generalizes that proposition for the 
relative case. 
Let {Cl,..., c,} = R- be the roots corresponding to B-. By definition, these 
are the characters of T appearing in the action of T on the tangent space of U- 
(so s = dira U-). They are nontrivial characters, and none of the ci is a power 
of oj for i  j (in particular, they are different). Moreover, for each ci there 
exists a unique additive subgroup G  c U- stable by T and such that T acts by 
multiplication by the character ci, in such a way that the multiplication morphism 
G  x .-- x G » - U- is an isomorphism of T-varieties. 
If  C Gs is a parabolic subgroup containing B, then, in particular,/çqU = H 
is a subgroup of U stable under the action of T by conjugation. Therefore, it 
suffices to study the structure of these subgroups. 
Definition 2.5. A cone is the spectrum of an N-graded algebra. 
One observes that a Z-graduation on an algebra A is equivalent to an action of 
algebras of the multiplicative group G on A. 
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Definition 2.6. We say that an action of Gr on Spec A is conic if the corresponding 
graduation on A is an N-graduation. 
Proposition 2.7. If G,, acts on an affine scheme X in a conic way and Y C X 
is a stable closed subscheme, then the action of G,, on Y is conic. 
Proof. hmnediate. [] 
Remark 2.8. The action of the maximal torus T on U- (with the above notation) 
is conic for certain one-parameter subgroups Gin -' T. 
Lemma 2.9. The systern of negative roots can be ordered in such a way that for 
each i _ s there exists a multiplicative subgroup G ) C T such that the restriction 
of the foot cj fo G ) is a nontrivial negative character for j > i and is the trivial 
character ]or j = i. 
Pro@ Using the additive notation for the characters of T, one has that R- C 
d 
X(T) = Z @ --. @ Z; thns, R- are vectors with negative coordinates. One then 
has to find linear forms w" X(T) --} Z (which correspond to multiplicative sub- 
groups of T) such that wi(aj) are negative for j > i and Cdi(Oi) = 0, for a certain 
order of the indices. By recurrence, it clearly suflïces to prove that for any subset 
J = {,...,} C R- there exists a reordering of the indices and a linear form 
w" X(T) -- Z such that w is negative on flj for j > 1 and c0() = 0. Since R- is 
contained in an open hall-plane of the corresponding rem vector space, any J C 
gencrates a convex cone that contains no full line. Let   J generate an extremal 
ray of that cone. Then there exist a linear form w vanishing at /, and strictly 
negative on ail other   J (since no negative roots are proportional); we may 
further assume that w is rational, and even integral. [] 
Corollary 2.10. Let us riz an or&ring of the roots 9iven by the precedin9 propo- 
sition. I] we denote U- = G  x ... x G , then 
 U( is closed in U- and stable under the action of T; 
 G ) acts in a conic way on U-, and the subscheme of the fixed points is 
Proposition 2.11. 
(1) Let H be an affine S-scheme in groups on which Gm acts in a conic way 
through automorphisms of S-schemes in groups. Then, there exists a unique retract 
p" H -- H ' of G-schemes, where H ' is the closed subscheme in groups of the 
fixed points of H by G,,. Moreover, p is a morphism of S-schemes in groups, and 
hence ker p - N is the unique subscheme in groups of H stable by G such that 
H = N ) H a,.. 
(2) If H is an affine S-scheme in groups on which G,- acts in a conic way 
through automorphisms of S-schemes in groups and H C H is a subscheme in 
groups stable by Gré, then G,, acts in a conic way on H and the restriction of the 
retract p: H  H a' to - is the corresponding retract p-: -  -H -a' c H a'". 
Pro@ (1) By the uniqueness of p it suffices to prove the statement locally on S. 
That is, we may assume S = Spec k with k a ring, and H --- Spec A, with A a Hopf 
k-algebra. 
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By hypothesis, A is a Hopf T-algebra, i.e., an N-graded Hopf k-algebra. Let us 
consider the Hopf subalgebra A0 C A of the elements of degree zero. Let I C A 
be the irrelevant ideal. One has that I is the idem of a subgroup of H over k and 
Ao  A/I as Hopf k-algebras. This means that the inchlsion Spec A/I C Spec A 
has the retract r0" Spec A --, Spec A0. Now, Spec A/I = H a-, since a point of 
H valued at B, h: A -- B (morphisnl of k-algebras), is a fixed point if and only 
if h is a morphisn of G,-algebras, where B is endowed with the trivial structure 
of Gm-algebra; in other words, if and only if it is a nlorphisnl of graded k-algebras 
with the trivial graduation on B (B = B0). But this is equivalent to saying that 
h(I) = 0, i.e., h factors through A/I or It is a point of Spec A/I valued at B. This 
yields the existence of p and proves that it is a morphism of schemes in groups. 
The uniqueness is given from the fact that if a morphism p" A/I --, A is a section 
of algebras and Gm-algebras of the projection r" A - A/I, then it maps into A0 
(since it is a morphism of G,-algebras) and, since it is a section of n, it must be 
the inverse of the isomorphism nlA °  Ao -- A/I. 
(2) It is immediate that the action is conic. For the second part, in the above 
reduction, the retract is given by the inclusion A0 C A: if A is the ring of H and A 
the ring of H, tllen the statement follows fronl the colnnlutativity of the diagranl 
A0(----- A 
0  

Corollary 2.12. With the notation of Corollary 2.10, one bas that U- is a sub- 
group of U-, and if is the kernel of the unique retract of groups and T-schemes 
 ÇOi -- 1 
Pi--1 U(_ l ---o a  

Pro@ (By recurrence on i). One has that U-_ is a subgroup of U-. Let us 
consider the subgroup G -) C T of Lemma 2.9. From Corollary 2.10 one has 
that (U-_)a -') = G '- and, by Proposition 2.11, there exists a unique retract 
p_" U-_  G « of G(-)-schemes which is, in addition, a norphism of groups. 
-- and hence ker p_  = U-- ; 
Therefore, it must coincide with the projection on ..« , 
in particular, UÇ is a subgroup of U- and p_ is a norphism of T-schemes. [] 

Proposition 2.13. 
of the form 

The subschemes in groups H of U stable by T are subschemes 

H = H x ... x H c U 
S S 
Oi 
with Hi C (G )s a subscheme in groups stable by homotheties. 
Pro@ In order to simplify the notation, let us assume that S = Spec k, i.e., H C 
U-, since the proof does not depend on the base S. Let us denote T = G  C T. 
By Corollary 2.10, U- and H are cones with respect to T, and the fixed points are 
respectively (U-) T = G  and H T = H  (U-) T = H ç G  = H. Moreover, 
the retract p: U- --, G  of T-schemes in groups restricts to the corresponding 
retract PlH" H -- H (Proposition 2.7). Thus, since kerp = U-, one concludes 
that H = H x H with H C G  and H C U-. Recurrently, one concludes. [] 
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Proposition 2.14. 
(1) If S is a connected scheme, then the subschemes in 9roups of U fiat over 
S and stable by T are constant; tbat is. they are all obtained from the subgroups of 
U- by tbe base change S -- Speç k. 
(2) If S is integral, then the subschemes in groups of U stable by T and fiat 
over S are precisely the ones with constant dimension and nilpotence degree along 
tbe fibres over S. 

Pro@ We may assume S = Spec A. By the preceding proposition, any subgroup H 
of U stable by T (without flatness conditions over S) has the form H1 x .-. x H, 
S S 
with Hi C (G*)s a stable subgroup by T. We may therefore assume U- = Ga = 
Spec k[Y] and H C (Ga)s stable by homotheties. In this case, H = Spec tA/Ii .Y 
i 
for certain ideals I, C A satisfyilg Iv C I C --- C I, C -.-. 
(1) If H is fiat over S, then the A-modules A/Ii are also fiat; thus, they are 
locally free with rank less than or equal to 1, i.e., either I = 0 or I, = A. It is then 
clear that H descends to U-. 
(2) Assume S is integral. If H C (G)s has constaut dimension 1 along the 
fibres, then H = (Ge)s, since the equality holds in the fibres over S. Therefore, 
(/i)0 = ç and hence Ii = 0 (since S is integral). If H has constant dimension 0 
along the fibres, then for each point x  S the nilpotence degree is the maximum 
index i such that p. D Ii (Px C A being t.he prime ideal of the functions that are 
zero at x). Since the nilpotence degree is constant along the fibres, one has that 
either (Ii)0 = S or (Ii)o = ; that is, either [i = 0 or [i = A for each i, and one 
concludes. [] 

Proof of Theorem 2.2. This follows from Lemma 2.4 and Proposition 2.14. [] 

3. TRIVIALITY OF THE ACTIONS WITH COMPLETE ORBITS 
Definition 3.1. Let X be a variety on which an algebraic group 1 acts and let 
x  X. We shall call the nilpotence degree of the orbit of x the nilpotence degree 
of its isotropy group: 
t?emark 3.2. In characteristic zero this notion is superfluous, since any group is 
smooth and hence the nilpotence degree of any orbit is 1. 
Theorem 3.3. Let  be an affine, smooth and connected al9ebraic group acting on 
a smooth variety X. Assume that the orbits of the action are complete and that they 
bave constant dimension and nilpotence de9ree. Then all the orbits are isomorphic 
as G-varieties to a given one Y and X oe Y x X/G as G-varieties, where the action 
of G on X/G is trivial. In particular. X/G is a geometric quotient of X by G. 
Pro@ Choose B, a Borel subgroup. Let X B be the reduced subscheme of the fixed 
points of X under the action of B. Then B fixes a unique point in any (-orbit 
in X. So the map 1 x X B -- X is surjective and its fiber at x  X e is G (i.e., 
the isotropy group of x), which is irreducible and of constant dimension (since the 
dimension of G/G is constant). Thus, X B is irreducible. 
Let us define b: (7 x X   X x X B by çS(g,x) = (g.x,x) and let P = 
b-l(Axs) C G x X , where Axs = X  C X x X  is the diagonal. It is a 
subscheme in groups of G x X  over X8; it contains B x X , and its fibre in each 
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point x E X B is the isotropy group of x that, by hypothesis, has constant dimen- 
sion and nilpotence degree. Since X B is connected and integral, one concludes, by 
Theorem 2.2, that it is constant:  =  × X B, with  C G a subgroup containing 
B. It thus follows that ç factors through an isomorphism G/-fi x X B -- X. [] 

4. INVARIANT INVERTIBLE SHEAVES 
Definition 4.1. Let X be a conlplete and homogeneous variety. An invertible 
sheaf/: E Pic (X) is said to be invariant if it is a fixed point of Pic (X) under the 
natural action of Aut°(x) in Pic (X). 
Remark 4.2. If X is a parabolic variety, then any invertible sheaf is invariant, since 
Picr«d(X) is a discrete k-scheme (because Pic°(X) = 0) and Aut°(x) is smooth 
aud connected. 
Proposition 4.3. Let X be a complete and homogeneous variety. If/:  Pic(X) 
is effective and invariant, then the complete linear system F(X,/:) has no base 
points, Aut°(x) acts by automorphisms on IF(H°(x,/:)*), and the natural mor- 
phism ¢£: X -- IF(H°(X,/:)*) is an Aut°(X)-morphism. 
Pro@ Let G = Aut°(x), and let m: G × X -- X be the action. Since /: is 
invariant, one has m*/: -/:c '/:, for some invertible sheaf/:c on G. Taking direct 
k 
images over G, one obtains 1 ® m*: (9a ® H°(x,/:) /: ® H°(x/:), and hence 
k k 
a morphism of G-schemes 
G x IF(H°(/:)) = IFa(Oa ® HO(/:)) ---" IFc(/:a ® HO(/:)) : G x IF(H°(/:)), 
k k 
where we bave denoted HO(/:) = H°(x,/:). Taking a fibre at each point of G, 
one obtains a morphisnl r: G --, Aut(IF(H°(X,/:))) = PGI(H°(X,/:)) such that 
for any rational point g  (7 and any section s  H°(X,/:) one has rg((s)) = (g's) C 
H°(x, g * /:) = H°(X,/:). Hence the transposed morphism 
satisfies çg((w)) = (g .w), for any g e (7 and w e H°(x,/:) *, where (g .w)(s) = 
w(g*s) for each s  H°(x,/:). It follows easily that ç is a morphism of groups (no- 
tice that a morphism between two smooth groups is a morphism of groups if and 
only if it is so for the rational points). Moreover, if for each point x  X we denote 
by w : H°(X,/:) --,/: - k the linear form that maps each section to its fibre at 
x, then (g.w) = (wa.). Therefore, the closed subset of base points is empty, since 
it is G-invariant, /: is effective, and X is homogeneous, and the natural lnorphism 
¢£: X --, IF(H°(X, £)*), defined by ¢£(x) = (w), is a G-morphism. [] 
Remark 4.4. This theorem implies the following well-known results: 
(1) IfX is an abelian variety, £  Pic°(X) is nontrivial if and only if H°(X, £) = 
0. 
(2) A complete and homogeneous variety X is parabolic if and only if Aut°(x) 
is an atfine group (and, in that case, it is semisimple of adjoint type). 
Definition 4.5. We shall soEv that a morphism r : X --, Y of smooth varieties is a 
quotient of X if 
(1) 7r is surjective (as a map), 
(2) r is submersive, i.e., a subset U C Y is open if and only if r-l(U) is open in 
X, 
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(3) O- = ker(Id ® 1 - 1 ® Id), with ld ® 1 - 1 ® ld: rr.Ox - rr.Ox ® rr.Ox the 
O 
natural morphism. 
We shall say that a quotient rr: X  Y is a.n invariant quotient if 
(4) there exists an action of Aut°(x) on Y such that rr is a Aut°(X)-morphism. 
This notion of quotient corresponds to the notion of quotient by an equivalence 
relation in the theory of schemes; tha.t is, it is wider than Mumford's notion of 
geolnetric quotient under the action of a group. 
Remark 4.6. If X is a homogeneous variety, then one inorphism r: X -- Y (where 
t" is a smooth variety) is an invariant quotient of X if and only if it is surjective 
and invariant (i.e., it satisfies the conditions (1) and (4)). Indeed if r is invariant, 
then the fibers are equidimensional and so r is fiat (X, Y are smooth varieties). 
Moreover, if r is surjective, then r is faithful fiat, and one concludes easily, by fiat 
descent, that r satisfies conditions (2) and (3). 
Definition 4.7. We shall say that two quotients r: X -* t', r'- X --, t" are 
equivalent if there exists an isomorphism ¢: Y -* Y such that r  = ¢ o r. 
Remark 4.8. Let X be a homogeneous variety. An invariant quotient r: X -, t" is 
determined, up to equivalence, by the fibre passing through a given point x  X, 
that is, by t.he subscheme n-(r(x)) C X. lndeed, if G = Aut°(X) and H is the 
isotropy group of x, then X = G/H, Y = G/H' (with H' the isotropy group of 
r(x)) and n-(r(x))= H'/H C G/H. 
Remark 4.9. The invariant quotients of X, up to equivalence, are partially ordered 
in the following way: (Y, r) _> (Y', r') if there exists a morphism h" t" --, I" of 
Aut°(X)-varieties such that r' = h o r. Moreover, they form an inverse system: 
if (Y,r), (Y,r) are two invariant quotients, there exists a third one, (Y,r), 
such that (Kr) _> (t,r), (I),r); it is enough to define Y as the image of the 
Aut°(X)-morphism n x r" X -- Y x I). Notice that if t and I are parabolic 
varieties (respectively, abelian varieties), then t" is parabolic (respectively, abelian). 
Lemma 4.10. Let X be a complete variety, £ an invertible sheaf on X and t) C 
V C H°(X. £) two linear systems without base points. Let r" X --, X C 1?(1/*) be 
the morphism induced by Vi, with X, = hn ri. There exists a morphism h: X -* 
X such that r = h o . Moreover, h is finite. 
Pro@ Let (V)0 C VÇ be the subspace incident with V» Let us consider the natural 
projection : F(lï)-ll((l)0) -- II(V*). Then r-(XçlP((l)0))is the subscheme 
of base points of 1, which is empty; hence  induces a morphism h" X --, X» 
The finiteness of h follows from the fact that it is an affine morphism, since h is 
affine. [] 

Proposition 4.11. Let X be a complete and homogeneous variety. There exists 
a parabolic (respectively, abelian) invariant quotient rr" X --, (X) satisfying the 
following universal property: for any parabolic (respectively, abelian) invariant quo- 
tient " X -- , there exists a unique morphism f: (X) --,  such that  = for. 
Moreover, there exists an effective and invariant invertible sheaf  on X such that 
T'(X) = Proj (,e H°( X' ') and the natural morphism 
= « Proj OH°(X,E ) = T'(X) 
nN 
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coincides with 

Proof. The invariant quotients are deternined by the fibre through a given point x, 
in such a way that the ortier of the quotients corresponds with the inclusion order 
of the fibres; that is, (Y,r) > (,W) if and only if r-Qr(x)) C w-l(=(x)). Since 
X is Noetherian, one concludes the existence of a maximal parabolic (respectively, 
abelian) quotient. 
For the second part, let £: be an ample invertible sheaf on P(X). Since 
is parabolic, £: is invariant, and so £: = r*£: is invariant too. Let us consider the 
parabolic quotient : X - P = Proj ()ne H°( X, £)" Olle has that P(X) = 
Proj(]EH°(P(X),- ') and H°(P(X),-) C H°(x,£). This defines a mor- 
phism h" P -- P(X) such that r = hot (Lenlma 4.10). Hence (P,) _> (P(X),Tc) 
and, by maximality, one concludes that (P,) = (T'(X),7c). [] 

Definition 4.12. The universal parabolic quotient 7c" X --> P(X) of the latter 
proposition is called the parabolic part of X. The universal abelian quotient 
7Cab" X --> Ab(X) is called the abelian part of X. 

5. STRUCTURE OF COMPLETE AND HOMOGENEOUS VARIETIES 
Lemma 5.1. Let G be a smooth and connected group. If G is a normal subgroup 
of a smooth and connected group G and G acts transitively on a variety Y, then the 
orbits of Y under the action of G are closed and conjugated by G (in particular, 
the orbits are isomorphic as G-schemes). In addition, there exists a fine quotient 
Y - Y/G. 
Proof. One has that Y "- G/H, as G-schemes. Since G is normal in G, it follows 
that G- H C G is a subgroup and it is clear that Y/G - G/G. H as G-schemes, in 
such a way that the morphism r: Y - Y/G corresponds to G/H -- G/G. H. One 
concludes easily. [] 
Theorem 5.2. If X is a complete and homogeneous variety over k, then: 
(1) X splits, in a unique way, as the direct product of a parabolic variety and an 
abelian variety, X = Y × A. These factors are canonically determined from X in the 
following way: A is the Albanese variety of X (that is, A -= Pic°(Pic°(X)), where 
Pic ° denotes the reduced connected component through the origin of the Picard 
scheme) and 7c1" X --> Y is the parabolic part of X. 
(2) X is projective, and (the connected and reduced component of) its automor- 
phism scheme splits, in a unique way, as a direct product of an abelian variety and 
a semisimple group of adjoint type. More precisely: if X = Y × A, with Y the 
parabolic part and A the abelian one, then Aut°(x) = Aut°(Y) × A and Aut°(Y) 
is a semisimple group of adjoint type. 
Proof. Let G = Aut°(x). Let G C G be the unique affine, smooth, connected 
and normal subgroup such that A = G/G is an abelian variety (Theorem 1.1). By 
Lemma 5.1, G acts on X with closed and isomorphic (as G-schenles) orbits. Then, 
by Theorem 3.3, X = G/P × X/G, and A = X/G is a quotient of G/G : A and 
hence an abelian variety. 
For the uniqueness of the decomposition X = Y × A, it is enough to determine 
canonically the projections of X onto the factors: 
(1) Let Pic ° be the reduced connected component through the origin of the 
Picard scheme of X, which is an abelian variety (Theorem 1.13(1)). It is known 
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that Pic°(Y) = 0 (Theorem 1.13(2)) and that Pic°(Pic°(A)) = A (Theorem 1.14), 
and this equality is canonical up to translations of A (that is, it is canonical once 
the origin is fixed). Moreover, Pic°(Y x A) = Pic°(t ") @ Pic°(A) (Theorem 1.15), 
and one thus has a natural morphism 
X ---, Pic°(Pic°(X))= Pic°(Pic°(Y) x Pic°(A))= Pic°(Pic°(A))= A, 
which is precisely the projectiou. In other words, A is the Albanese variety of X, 
aud the projection X  A is the natural morphism X --, Alb(X). 
(2) In order to sec that rl : X --, " is the parabolic part of X, it is enough to sec 
that (Y,h) >_ (7)(X),r), i.e., that the fibre of a point Y0 E t" by rl, Y0 x A C X, 
is contaiued lu some fibre of r (Remark 4.8). By Proposition 4.11, r: X 
is the morphisnl induced by an effective invariant invertible sheaf/2 on X; hence 
it suflïces t.o prove that /2 is trivial on Y0 x A. But, since A = Y0 x A is an 
abelian variety, mly effective invariant invertible sheaf is trivial (Remark 4.4 (1)). 
The restriction of /2 to A is effective because H°(x,/2) ¢ 0 has no base points 
(Proposition 4.3). 
For the second part of the statement, it is enough to observe that. siuce the 
projections onto the parabolic aud abeliau parts are invariant quotients, any au- 
tomorphism induces an automorphisnl of each factor, and conversely. The test is 
iminediate. [] 

Corollary 5.3. Any group extension of an abelian variety by a semisimple group 
of adjoint type is trivial. 

Pro@ Let A be an abelian variety, G a semisimple group of adjoint type, and let. 
0-+ G G ,4- 0 

be a group extension. Let B C G be a Borel subgroup of G and X = G/B. 
Since G is a normal subgroup of G, it follows eusily that the orbits of X under the 
action of G are all isomorphic to G/B. Hence, by Theorem 3.3, x = G/B x A 
(since G/G = A) and G c Aut°(x) = Aut°(G/B) x Aut°(A) = Aut°(G/B) x A. 
Let h" Aut°(G/B) x A --, Aut°(G/B) be the natural projection. Then h(G) = 
Aut°(G/B) = G (Demazure [4]). It follows that h" G --, G is a retract of groups, 
and hence G is the trivial extension. [] 

Definition 5.4. The parabolic type of a homogeneous variety X is the class, mod- 
ulo automorphisms of Aut°(x), of the isotropy group of any point of the variety 
under the action of its group of automorphisms (more properbç the reduced con- 
nected component of its group of automorphisms). 

Remark 5.5. From the preceding theorem, the isotropy group of any point x of 
a complete homogeneous variety X is a parabolic subgroup of the affine part of 
its group of automorphisms. Therefore, the parabolic type of the variety coincides 
with the class of its parabolic part modulo isomorphisms. 

Theorem 5.6. Two complete and homogeneous varieties are isomorphic if and only 
if their groups of automorphisms are isomorphic and they bave the same parabolic 
type. In partzcular, the group of automorphisms classifies the variety, once ifs 
parabolic type bas been given. In characteristic zero, there exist af most a finite 
number of complete homogeneous varieties with a given group of automorphisms. 
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Given a complete and homogeneous variety X = Y x A, let us denote by 
Autli'(X) the maximuin affine, normal, smooth and connected algebraic sub- 
group of Aut°(x) (as usual, Aut°(x) denotes the reduced connected component 
of the group of automorphisms of X), and let AutAb(x) = Aut°(x)/AutUn(x) = 
Alb(X). 
Theorem 5.7 (Classification of complete homogeneous varieties). The classifi- 
cation of complete homogeneous varieties is equivalent to the classification of the 
triplets (A, G, [PI), where A is an abelian variety, G is a semisimple group of adjoint 
type, and [PI is a parabolic type of G such that G = Aut°(G/P). The correspon- 
dance is 
X  (Alb(X),Autli'(X),[P]), 
A x G/P - (A. G, [PI), 
where Px C AutUn(X) is the isotropy group of a,y point x E X and [Pe] de, ores 
ifs class modulo automorphzsms of AutUn(x). 

6. CLASSIFICATION OF PARABOLIC VARIETIES 
To conclude, we are going to give the classification of the parabolic varieties from 
the following results: 
 The classification of the parabolic subgroups given by Wenzel in [12] (in char- 
acteristic different from 2 aud 3). 
 The determilmtion (given by Demazure in [4]) of the pairs P C G, where G is a 
simple group of adjoint type and P Ç G is a reduced parabolic subgroup such that 
G : Aut°(G/P). The pairs satisf.ving this condition are called non-exceptional. 
Demazure proves that the exceptional pairs are the following ones: 
(1) G = SO2/+ (k) and G/P t he variety t.hat parameterizes the totally isotropic 
subspaces ç[ C k 21+ (with 2/+1 _> 5). In this case Aut°(G/P) - PSO2I+2. 
(2) G : SP2/(k ) and G/P : ]P2l- the variety that parmneterizes t.he lines of 
k 21. In this case Aut°(P21_) - PG121(k). 
(3) G = the simple group of adjoint type with semisimple rank 2 and type G; 
that is, it is the group of automorphisms of an algebra of octonions ç. Let 
 C  be the hyperplane of the pure octonions and G/P the variety of 
isotropic lines of ç. Then G/P is isomorphic to a projective quadric of 
dimension 5, and hence, Aut°(G/P) = PSO6(k). 
Theorem 6.1. A complete homogeneous variety X splits as a direct product of two 
varieties X =  x }'2 if and only if the group of automorphisms Aut°(x) splits as 
a direct product of two groups Aut°(x) = G x G2. Moreover, one bas: 
(1) Y, Y2 are complete and homogeneous varieties. 
(2) ç(x)= ç(i) × 
(3) Ab(X)= Ab(Y) x Ab(Y). 
(4) Aut°(x) = Aut°(Y) x Aut°(Y). 
Pro@ Assume that X = Y x Y2- Since Y _ t' x y  X for any closed point y2 E 
Y2, Y is projective. Analogously, I is projective. Moreover, they are both reduced 
and connected, since the product Iq x Y2 = X is so. In particular, Pic°(X) = 
Pic°(Y) x Pic°(Y). It follows that Ab(X) = Ab(Y) x Ab(Y) and the morphism 
tz: X --, Ab(X) is precisely/z = /z x/z2: X = Y x I  Ab(t]) x Ab(Y2). On 
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the other haud, the fibre by # of any point x = (Yl, Y2) is P(X) = F1 x F2, where 
F1 C }, F2 C Y2 are the fibres by #1 and P2 of the points Yl and y2, respectively. 
Analogously, one proves that F1, F2 are projective and connected varieties. Let 
£1 be an effective ample invertible sheaf on F1. Then £1 = £1 ®OF is an invertible 
k 
sheaf on P(X), and hence invariant (since P(X) is a parabolic variety). The natural 
morphism 

P(X) -- Proj O H°(P(X)'£î) = Proj O H°(F'î) = F1 

is an invariant quotient, and it coincides with the projection on the first factor. 
One concludes that F1, F2 are Aut o (P(X))-varieties and Aut o (P(X)) = Aut o (F1) x 
Aut°(F2). lu particular, oue bas that 

Aut°(x) = Aut°(P(X)) x Ab(X) 
= (Aut°(F1) x Ab(YI)) x (Aut°(F2) x Ab(Y2)) = (71 x (72; 

that is, Aut°(x) splits as a direct product of groups in such a way that G1 acts 
trivially on Y2 and G2 acts trivially on Y1- So Aut°(x) = G1 x G2 C Aut°(]t) x 
Aut°(Y2) C Aut°(x), and then GI = Aut°()]) and G2 = Aut°(Y2). 
Conversely, if Aut°(x) 
Gî b x G2 Ab. In particular, Gï is a normal subgroup of Aut°(x), and this is an 
affine smooth and connected group. By Lemma 5.1. X = F1 x X/Gï, and the 
projections are Aut°(X)-morphisms. FI is an orbit under the action of Gï; hence 
Gï n is semisimple of adjoint type, F1 = Gï/P1 and the projection X -* F1 is 
an Aut°(X)-morphism. Therefore, Gî b × G2 induces a group of automorphisms 
of F1 that commute with 
Auta-«(Gl /P1)«d = 
(Naï,(P1)/P1)«d = 0, where Na,,P1 ) is the normalizer of Pi in Gï, because 
if g  GI ' normalizes PI, then it normalizes (P1)«d, and hence g G (Pl)«d C PI, 
since Na(P) = P for any reduced parabolic subgroup P of an affine smooth 
and connected group G. Consequently, Aut°(F1) = Gï and Aut°(X/Gï) = 
Gî b x G2. Repeating the saine argument with G  and X/Gï, one obtains that 
X/Gï -= F2 x X/(Gï x G ') = F2 x Ab(X), where the projections are in- 
variant quotients, Aut°(X/Gï ) = Aut°(F2) x Aut°(Ab(X)), Aut°(F2) = G ' 
and Aut°(Ab(X)) = AutAb(x) = Gî b × G Ab. Regrouping the factors, one has 
X = (F1 × Gî b) × (F2 × G2 Ab) - Y1 x 1 and Aut°(Y1) : G1, Aut°(Y2) -= G2. [] 

Definition 6.2. We say that a homogeneous variety X is indecomposable if it is 
hot the product of two varieties of dimension greater than zero. 

Corollary 6.3. A parabolic variety X is indecomposable if and only if its auto- 
morphism group is a simple group of adjoint type. 

Corollary 6.4. Any parabolic variety P splits uniquely (up to permutation of the 
factors) as a dzrect product of indecomposable parabolic varieties, 

Proof. This follows from Theorem 6.1 and from the existence and uniqueness of the 
decomposition of an affine semisimple group of adjoint type as a product of simple 
groups. [] 
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Remark 6.5. Fom Theorem 5.7 and Corollary 6.4, the classification of homogeneous 
varieties is reduced to the classification of abelian varieties and the classification of 
parabolic varieties with simple (of adjoint type) group of automorphisms. 
Let G be an aflïne, simple, algebraic group of adjoint type. Let/3 = G/B. Then 
G = Aut°(/3) (see [4]), and any parabolic G-variety 79 is an invariant quotient 
/3 -* 79 (in a unique way up to translations by elements of G). The problem is to 
determine the parabolic G-varieties 79 satisfying Aut°(79) - G. 
First of all, let us specify what are the parabolic G-varieties, up to isomorphisms 
of varieties. Let us consider the associated parabolic varieties 791 = G/PI,..., 79s 
G/Ps, where P1,--., Ps are the maxinml reduced parabolic subgroups containing 
B. 
Given two parabolic G-varieties n" /3 -, 79 and n': /3 -, 79', we shall denote by 
79 * 79' the parabolic G-variety image of the G-morphism r x r': B -* 79 x 79'. 
Remark 6.6. If 79 = G/P and 7 9' = G/P' with P. P C G parabolic subgroups 
containing B, then 79 * 79  = G/Pç P'. Moreover, 79 * 79' = sup(79, 79), with respect 
to the order of the parabolic G-varieties. 
Assume char(k) = p > 0. Given a variety X and a natural number n E N, 
we shall dcnote by X['q the scheme whose underlying topological space is X and 
pn 
whose structural sheaf is the subsheaf 0 x C Ox. One has a natural morphism of 
schemes F " X -* X [1 (F being the Frobenius morphism). 
If G is a scheme in groups, then G ['q is a scheme in groups and F 
is a morphism of groups. We shall denote G, = ker F', which is a subscheme in 
groups, finite and local. 
Remark 6.7. It is easy to sec that if 79 is a parabolic G-variety, then 79 ['q is a 
parabolic G-variety too, and Aut°(79 M) = Aut°(79) M. 
Theorem 6.8. Assume that 0 < char(k) ¢ 2, 3. For each parabolic G-variety 7 9, 
there exist unique indices 1 <_ il < .  < if <_ s and exponents n , . . . , nf  N such 
that 
p = p!',l ,..., 
Moreover, AutO(79) = G if and only if nh = 0 for some 1 < h <_ r and 79ij is 
non-exceptional for some 1 <_ j <_ r. That is, G --* AutO(79) is hot an isomorphism 
if and only if either n,..., nf > 0 or P is maximal and P C G is an exceptional 
pair. 
Pro@ With the notation of [12], one has that each parabolic subgroup P of G 
containing B can be expressed in a unique way as P = Çe,v P,,,,, where S is 
the basis of the foot system of G corresponding to T C B, P, is the maxinml 
parabolic subgroup such that the foot system of (P, T) does hot contain -/5i, and 
P,,, = G,  P if ni # oc, and P, = G. With our notation, G/P, = 
G/P,, = 79j,d if ni # 0% and G/P, = Speck; then 79 = G/P = 79['11 *--. * 
79 ''1, where 79J 1 = Spec k, i.e., it is a factor that can be suppressed. 
For the second part, one observes that G -* Aut o (79) is injective if and only if 
inf(nl,..., n) = 0, since, on the contrary, it factors through G [1]. Moreover, it 
is surjective if and only if there exists a morphism r: AutO(79) -. G M that co- 
incides with F' over G. Indeed, if zr exists, then (ker zr) ° is a normal subgroup 
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of Aut°(P), and (kerTr) ° ç G = {e}. Hence, since Aut°(P) is simple, one con- 
cludes that (kerTr) ° = {e}, and then dimAut°(P) = dimG [1 = dimG: hence 
G = Aut°(P). Now, if (G, Pi) is non-exceptional, then Aut°(Pi) = G. So (by the 
following lenmm) one has Aut°(P) - Aut°(P} d) = Aut°(P)[d = G [ml and one 
concludes. Conversely, if P is maximal and the pair P C G is exceptional, then 
G Ç Aut°(P). [] 

Lemma 6.9. With the notation of the preceding theorem, the projections 
are invariant quotients. 

Pro@ Let us consider the G-morphism 7ri" P - pi], let  be a very ample 
invertible sheaf on pl and £ = 7r'. Let : P --,  be the invariant quotient 
defined by the complete linear system H°(P, £) (Proposition 4.3). It is clear that 
 factors through a morphism h"  --, P'] and h* is a very ample invertible 
sheaf on ; hence h"  - pm] is a finite morphism with connected fibers (sinee 
these fibers are quotients of parabolic subgroups of G). Then  = p}l and. by the 
preceding theorem (first part),  _> n. It is clear that P = Pl'].---.P]*-.-*P[]; 
hence, again by the first part of the preceding theorem, n = ni; that is,  = pn] 
and ri = h is an invariant quotient. [] 

Let G be a simple algebraic group (of adjoint type) and :DG its Dynkin dia- 
gram. It is a connected graph (the Coxeter graph, whose vertices are the elements 
of a basis of roots of G), where each edge bas a weight m - 1.2, 3. One has 
that Aut(G)/Inn(G) = Aut(:DG), i.e., the automorphisms of G, modulo inner au- 
tomorphisms, are the automorphisms of the graph :DG leaving the weights invari- 
ant. Moreover, one has that Aut G = Inn(G)  Aut(:DG), where the identification 
Aut(:DG) C Aut G is given by the automorphisms of G that leave invariant a given 
Borel subgroup B and a given maximal torus T C B. Therefore, Aut(:DG) acts on 
the set of parabolic subgroups containing B. The action is as follows: the vertices 
of :DG correspond with the elements of a basis $ of roots of G, and these correspond 
with the reduced and maximal parabolic subgroups of G containing B, P,..., P. 
On the other hand, the parabolic subgroups of G containing B correspond biuni- 
vocally with the functions ¢: $ - N U ec, i.e., with N* x .-. x N* = N *, where 
N* -- NU c and Aut(:DG) acts on $. Finally, for G fixed, the parabolic G-varieties, 
modulo isomorphisms of varieties, correspond with the parabolic subgroups of G, 
modulo automorphisms of G, i.e., with the parabolic subgroups of G containing B, 
modulo Aut(:DG). 
In conclusion, in characteristic different from 2 and 3, one has: 
Theorem 6.10 (Classification of indecomposable parabolic varieties). Let G be a 
simple algebraic group (of adjoint type), $ the basis of roots of G corresponding to 
T C B C G, and :DG its Dynkin diagram. The set of parabolic varieties whose auto- 
morphism group is isomorphic to G, modulo isomorphisms of varieties, is identified 
wth the subset of the set 
{Parabolic G- Varieties}/ ,  N*/Aut(:Da) 
formed by the classes of elements (n,..., n)  N *s such that ni = 0 for some i 
and n i 7 oc for some j such that Pi C G is non-exceptional. 
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A PATH-TRANSFORMATION FOR RANDOM WALKS 
AND THE ROBINSON-SCHENSTED CORRESPONDENCE 

NEIL O'CONNELL 

ABSTRACT. The author and Marc Yor recently introduced a path-transfor- 
mation G (k) with the property that, for X belonging to a certain class of 
random walks on Z_, the transformed walk G(k)(X) has the same law as the 
original walk conditioned never to exit the Weyl chamber {x : Xl _< --- <_ xk}. 
In this paper, we show that G (k) is closely related to the Robinson-Schensted 
algorithm, and use this connection to give a new proof of the above repre- 
sentation theorem. The new proof is valid for a larger class of random walks 
and yields additional information about the joint law of X and G(k)(X). The 
corresponding results for the Brownian model are recovered by Donsker's theo- 
rem. These are connected with Hermitian Brownian motion and the Gaussian 
Unitary Ensemble of random matrix theory. The connection we make between 
the path-transformation G (k) and the Robinson-Schensted algorithm also pro- 
vides a new formula and interpretation for the latter. This can be used to 
study properties of the Robinson-Schensted algorithm and, moreover, extends 
easily to a continuous setting. 

1. INTRODUCTION AND SUMMARY 
For k >_ 2 denote the set of probability distributions on (1 ..... k} by k. Let 
(m, m > 1) be a sequence of independent random variables with common distri- 
bution p C k and, for 1 < i < k, n > 0, set 
(1) x() = 
If Pi < "'" < Pk, there is a positive probability that the random walk X -- 
(X1,..., X) never exits the Weyl chamber 
(2) V : {x  ]k . Xl _ "'"  Xk}; 
this is easily verified using, for example, the concentration inequality 
where c() and K are finite positive constants. 
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In [36], a certain path-transforlnation G (k) was introduced with the property 
that: 

Theorem 1.1. The law of trie transformed walk G(k)(X) is the saine, assumzng 
Pi < "'" < Pk, as that of the origitml walk X conditioned never fo exit 
We will recall the dcfinition of G (k) in Section 2 below. 
This was motivatcd by a desire to find a multi-dimensional generalisation of 
Pitman's representation for the three-dimensional Bessel process [37], and to un- 
derstand SOlne striking Colmections which were recently discovered by Baik, Deift 
and .lohansstm [4], Baryshnikov [5] and Gravner, Tracy and Vidom [21], between 
oriented percolation and random matrices. For more background on this, sec [33]. 
The proof of Theorem I.I given in [36] llSeS certain symmetry and reversibility 
properties of M/M/1 queues in series; consequently, the transformation G (k) has a 
"queueing-theoret ic" interpretation. 
In this paper we will show that the path-transformation G (k) is closely related to 
the Robinson-Schensted correspondence. More precisel3; if ,(n) = (,1 (n) _> ---_> 
A(n)) denotes the shape of the ¥OUllg tableaux obtained, when one applies the 
Flobinson-Schensted algorithm with cohlmn-insertion, to the random word ç --- 
then (for any realisation of X) 
(a(x))() = (() .... ,,(.)). 
Immediately, this yields a new representation and formula for the 1Robinson- 
Schensted algorithln, and this formula has a queueing interpretation. We will use 
this representation to recover known, and perhaps not-so-well-known, properties of 
the 1Robinson-Schensted algorithm. 
Given this connection, Theorem 1. I can now be interpreted as a statement about 
the evolution of the shape )(n) of a certain randomlv growing Young tableau. 
We give a direct proof of this result using properties of the [/obinson-Schensted 
correspondence. This also vields more information about the joint law of X and 
G()(X), and dispenses with the condition p < ... < pk. 
As in [36], the corresponding results for the Brownian motion model can be 
recovered by Donsker's theorem. The path-transformation G () extends naturally 
to a continuous setting and, given the COllllection with the Pt.obinson-Schensted 
algorithm, the continuous version can now be regarded as a natural extension of 
the 1Robinson-Schensted algorithm to a contilmous setting. As discussed in [36], 
the results for Brownian luotion bave an interpretation in ralldolll matrix theory. 
In particular, Theorem 1.1 yields a representation for the eigenvalue process as- 
sociated with Hermitian Brownian motion as a certain pat.h-transforlnation (the 
continuous analogue of Ç()) applied to a standard Brownian motion. The new 
results presented in this paper also yield new results in this context. This random 
matrix connection comes from the well-known fact that the eigenvalue process as- 
sociated with Hermitian Brownian lnotion can be interpreted as Brownian motion 
conditioned never to exit the Weyl chamber IV. \Ve remark that a similar rep- 
resentation for the eigenvalues of Hermitian Brownian motion was independently 
obtained by Bougerol and Jeulin [11], in a more general context, by completely 
different methods. 
The outline of the paper is as follows. In the next section we recall the definition 
of G () and record some of its properties. In section 3, we make the connection with 
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the Robinson-Schensted algorithm, aud briefly consider some immediate implica- 
tions of this connection. A worked example is presented in section 4. In section 5, 
we record some properties of the conditioned walk of Theorem 1.1 and extend its 
definition beyond the case/91 < --- < Pk. In section 6, we prove a generalisation of 
Theorenl 1.1, in the context of Young tableaux, using properties of the RS corre- 
spondence. In section 7, we define a contilmous version of the path-transformation 
and prescrit the "Poissonized'" allalogues of the results of the previous section. In 
section 8, we present the correspondi,lg results for the Brownian model, and briefly 
discuss the connection with rmldom ,uatrices. Au application in queueing theory 
is preseuted iu section 9, and we conclude the paper with some remarks iii section 
10. 
Some notation: Let b = {el,..., ek} denote the standa,'d basis elements in IR k. 
For x,y E N we will write x y = xï ...x ', xy = (xly,...,xky), Ixl = -ixi 
and define x* E lR_î_ by x' = x-i+l. Denote the origin in IR  by o. 
Acknowledgements. Thanks to Frmwois Baccelli, Phillipe Biane, Phillipe Bou- 
gerol and Marc Yor for many helpflfl raid ilhmfinating discussions on these topics. 
This research was partly carried out during a visit, fimdcd by the (ï',NIS, to the 
Laboratoire de Probabilités, Université Paris 6, aud partly at the ENS, thanks to 
financial support of INRIA. 

2. THE PATH-TRANSFORMATION 

The support of the random walk X, which we denote by Plk, consists of paths 
x" Z+ --+ Z with x(0) = 0 and, for each n > 0, x(n)-x(n- 1)  b. Let I1[, v denote 
the subset of those paths taking values in W. It is convenient to introduce a, lother 
set Ak of paths x" Z+ --, Z_î_ with x(0) = 0 and x() - x(n- 1) e {0, ci ..... ek}, 
for each n > 0. 
For x, y  Al, define x/x y  A1 and x v y E A1 by 

(3) 

(x ix y)(n) = rein Ix(m) + y(n) - y(m)] 
0<rn<n 

and 
(4) (xvy)(n) = max [z(m) + y(n) - y(m)]. 
0<rn<rz 
The operations /x and v are not associative in general. Unless otherwise delineated 
by parentheses, the default order of operations is from left to right; for exa, nple, 
when we write z/x y/x z, we mean (x/x y)/x z. 
The mappings G () : Ak --, A are defined as follows. Set 
(5) G(2)(x,y) = (x/xy, yvx) 
and, for k > 2, 
G()(x,... ,x) = (xl /x/ .../x, 
(6) a(k-)(x v x, x3 v (Zl  xD,..., x v (Xl  .--  X-l))). 
Note that G () " H --, H '. 
We will now give an alternative definition of G (k) which will be useful for making 
the connection with the Robinson-Schensted correspondence. 
Occasionally, we will suppress the dependence of functions on x, when the context 
is clear: for example, we may write ((k) instead of G () (x), and so on. 
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(7) 
and 
(s) 

For k _> 2, define maps D (k)  Ak -- Aa and T ()  Aa -- Aa-1 by 
D(k)(x) - (XI,Xl AX2,...,XI A -'" AXk) 

T(k)(x) = (x2 VXl,X3 v (Xl AX2),---,Xk V(Xl 5 "'" 5k-1)). 
For notational convenience, let D () be the identity transformation 
Note that the above definition is recursive: for i  2, 
= D() 
(9) D k) i--1  Zi 
and 
o(k) 
(10) T}){ = Zi V i--1" 
Alternatively, we can write 
(11) (() T}2{) = a()(o}{ ) 
For each x  Ak, considcr the triangul array of sequences d (')  Ak-i+l, 

1 < i < k, defined as follows. Set 
d () = D()(x), 
d (2) = D(k-)(t()), 
and so on; for i _< k, 

t (i) = T(k)(x), 
t(2) = T(k-1)(t(1)), 

d(O = D(k-i+l)(t(i-1)), 

and for i _< k - 1, 
t(O = T(k-i+)(t(i-)). 
Recalling the definition of G (k) given earlier, we see that 
(12) G (k) -= (d(kl),... ,dk)). 
Note also that, for each i _< k, 
(13) G(')(Xl,... ,Xi) = (dl),..., di)). 
We will conclude this section by recording some useful properties and interpre- 
tations of the operations /x and v, and of the path-transformation G (k), for later 
reference. We defer the proofs: these will be given in the appendix. 
The following notation for increments of paths will be useful: for x E Ak and 
l _> n, set x(n, l) = x(1) - x(n). 
The operations A and v have a queueing-theoretic interpretation, which we 
will make strong use of when we make the connection with the Robinson-Schensted 
correspondence in the next section. For more general discussions on "rein-plus 
algebra" and queueing networks, see [1]. 
Suppose (x, y) E 1-[2, and consider a simple queue which evolves as follows. At 
each time n, either x(n)- x(n- 1) = 1 and y(n)- y(n- 1) = 0, in which case a 
new customer arrives at the queue, or x(n) - x(n - 1) = 0 and y(n) - y(n - 1) = 1, 
in which case, if the queue is not empty, a customer departs (otherwise nothing 
happens). The number of customers remaining in the queue at time n, which we 
denote by q(n), satisfies the Lindley recursion 
(14) q(n) = max{q(n- 1) + e(n), 0}, 
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where e(n) = x(n) - x(n - 1) - y(n) + y(n - 1). Iterating (14), we obtain 
(15) q(n)= max y(m,n)]. 
0<m<n[X(,) -- 
Thus, the mlmber of customers d(n) to depart up to and inchlding time  is given 
by 
(16) d(n) = x(n) - q(n) = (x  y)(n). 
We also have 
(17) t(n) := x(n) + u(n) = (yvx)(n), 
where 
() = y() - d() 
is the number of times m  n that y(m) -y(m- 1) = 1 and q(m - 1) = 0; in 
the language of queueing theory, u(n) is the number of "unused services" up to 
and including time n. (For this queue we refer to the points of incree of y 
"services".) 
Lemma 2.1. For (x, y) 
(18) xy+yvx =x + y 
d, il ie[x() - (x  v)()] = 0. 
x(n)-(xy)(n) = nmx [x(m,n)-y(m,n)] 
0<m<n 
= mx[(x)(,t)- (v*)(,,)]. 
In paicular, writing G (2)  G (2) (x, y), we bave 
(x(), ()) = (=)() + F(=) ((=)(,),  e ), 
(19) 
whe F ()    Z  is defined on 
 = 
n0 
 F(=)(z) = (Z(z),-M(z)). 
In the queueing context described above, Lemma 2.1 states that x + y = d + 
and, if min q(1) = O, 
(20) () = x[d(, ) - t(, ,)]. 
The first identity is readily verified. The formula for q(n) in terres of the future 
increments of d and t follows from the time-reversal symmetry in the dynamics of 
the system: this formula is the dual of (15). When time is reversed, the roles played 
 (x, y) and (d, t) are interchanged. This symmetry is at the heart of the proof of 
Theorem 1.1 given in [36], where it is considered in an equilibrium context. 
Note that. if we set z = y - x and s(n) = maxo<< z(m), then 
yvx - x y = 2s - z 
and (20) is equivMent to the well-known identity 
s(n) = min[2s(/)- z(/)]. 
This is familiar in the context of Pitman's representation for the three-dimensional 
Bessel process. Observe that the statement of Theorem 1.1 in the case k = 2 is 
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equivalent to the following discrete version of Pitnmn's theorem: if {Z(n),n >_ 0} 
is a simple random walk on Z with positive drift, started at 0, and we set S(n) = 
max0<< Z(n), then 2S - Z has the saine law as that of Z conditioned to stay 
nomwgative. The usual statement of Pitman's theorem tan be recovered ri'oto 
Theorem 8.1 below. 
Lenmm 2.1 bas the following generalisation: 
Lemma 2.2. For x  Ak, wting G ()  G()(x), we bave 
(2) 
and, 
-dj+)(1)=O. for l 5j < i < k (c). 
(22) «(n) = a(k)(n) + F (k) (a(k)(..l), l  n), 
where the function F () will be defined in the pro@ 
As we remarked earlier, thc opcrations and v are hot associative. The follow- 
ing identities are usefld for manipulating complex combinations of these operations. 
For (a, b, c)  A3 we have 
av (cb) v (bvc) = avbvc 

Lemma 2.3. 
(23) 
and 
(24) 

a/x (cvb)/x (bA) = a/xb/xc. 
For example, (23) immediately yields 
Lemma 2.4. For x (5 Ak, G( k)(x) = .rk v --- v xl. 

3. CONNECTION ,VITH THE PtOBINSON-SCHENSTED ALGORITHM 

We refer the reader to the books of Fulton [18] and Stanley [40] for detailed 
discussions on the Robinson-Schensted algorithm and its properties. The stan- 
dard Robinson-Schensted algorithm takes a word w = al...a, (5 {1.2,...,k} ' 
and proceeds, bi}' "row-inserting" the numbers al, then a2, and so on, to con- 
struct a senistandard tableau P(w) associated with w, of size n with entries from 
the set {1, 2 ..... k}. If one also maintains a "recording tableau" Q(w), which is 
a standard tableau of size n, the mapping from words {1, 2 .... , k} ' to pairs of 
senfistandard and standard tableaux of size n, the semistandard tableau having 
entries from {1, 2,..., k} and both having the saine shape, is a bijection: this is 
the Robinson-Schensted correspondence. One tan also do all of the above using 
"column-insertion" instead of row-insertion to construct the semistandard tableau, 
but still maintaining a recording tableau, and the resulting map is also a bijec- 
tion. Column and row insertion are not the smne thing, but they are related in 
the following way: the semistandard tableau obtained by applying the Robinson- 
Schensted algorithm, with column-insertion, to the word ol "--an is the saine as the 
one obtained bi}' applying the Robinson-Schensted algorithm, with row-insertion, to 
the reversed word a,...al. The standard tableaux obtained in each case are also 
relate& but we do not need this and refer the reader to [18] for details. 
Fix x (5 Hk, let d (i) (5 Ak-+l, 1 < i < k, be the corresponding triangular arrav 
of sequences defined in the previous section. 
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For each n, construct a senfistandard Young tableau as follows. In the first row, 
put 
, (1) ' ...dï ) k' 
dï)() ls, dï)(n)-d (,) 2s, (,)-dï-1)(n) s; 
in the second row, put 
d')(,) 2's, d2)(n) - d')(,) 3's, ... ,d-')(n) - d-2)(n) k's, 
aud so on. In the final row, there are just d(l)(,) k's. Denote this tableau by r(n). 

For example, if k = 3 and 
d) (7) 3() 
 (7) d(3')(7) 2 2 1 
(25) _,3()(7) d)(7) = 3 2 
dï)(7) 

then the corresponding senfistandard tableml "r(7) is 
1 1 --[- 
(26) 2 2 

Let ara be the sequence defined by ara = i whenever 

.r(m) -- x(m -- 1) = ci. 

Theorem 3.1. The semistandard tableau "r(n) is precisely the one that is ob- 
tained when one applies the Robinson-Schensted algorithm, with column inser- 
tion, to th« word ai.. "an. In particular, if l(n) d«not«s th« shap« of r(n). thon 
(n)* = 

Pro@ It will sutïqce to describe how the mapping G (k) acts on a typical element of 
Ilk, from an algorithmic point of view. 
For each k _> 2, the maps D (k) " Ak -- Ak and T (k)  Ak -- Ak-i cau be defined 
as follows. Fix x  1-lk and set d = D (k) (x), t = T (k) (x). Set d(0) = t(0) = 0, and 
define the sequences d(n) and t(n) inductively on n. Suppose x(n)- x(n- 1) = ei; 
that is, a, = i. We need to treat the cases i = 1 and i = k separately. 
Suppose i = 1. Then we set d(n) = d(n - 1) + e and t(n) = t(n - 1) + el. 
If i = k, and dk(n-- 1) < dk-(n-- 1), we set d(n) = d(n- 1) +ek and 
t(n) = t(n- 1). 
If i = k, and d(n- 1) = d-l(n - 1), we set d(n) = d(n- 1) and t(n) = 
t(n - 1) + e«. 
Nowsuppose 1 < i < k. Ifd()-l) < d_()-l), set d(n) = d(n- 1)+ci 
and t(n) = t(n - 1) + ci; if di(n - 1) = di-(n - 1), set d(n) = d(n - 1) and 
t(n) = t(n - 1) + Ci-l. Recall that D (1) is the identity transformation. 
In queueing language, we have just constructed a series of k queues in tandem. 
Initially there are infinitely many customers in the first queue and t.he other queues 
are all empty. At each time n, if x(n+ 1)-x(n) = 1 (or, equivalently, a, = i) there 
is a "service event" at the i th queue; if this queue is hot empty a customer departs 
from it and, if i < k, joins the (i + 1) th queue. The number of departures from the 
i th queue up to and including time n is given by d(n) and t(n) = di(n) + ui(n), 
where u(n) is the number of "unused services" at the (i + 1) th queue up to and 
including time n. Recalling the queueing-theoretic interpretations of /x and v, we 
sec that d = Xl/x    /x x and t = x v d-l. 
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Now fix x ff 1-Ik, and recall the definition of the triangular array of sequences 

d (i) E Ak-+l, 1 < i < k. Set 
d(U = D(k)(x), 
d () = D(k-1)(t()), 
and so on; for i < k, 

and lori<k-i, 

t () = T(a)(x), 
t () = T(a-1)(t()), 
d (i) = D(-i+ ) (t(i-1)), 
t () = T(k-i+U(t(i-)). 

Here we have constructed a "series of queues in series", the entire system "driven" 
by x. This is represented in Figure 1 for the case k = 3. 

   000 ] x 

FIGURE 1. The series of qllelles in series (k = 3) 

The first series of queues is just the one described above: there are k queues in 
the series, and initially queues 2 thru k are empty and the first queue bas infinitely 
many customers; whenever xi increases by one there is a service at the i th queue 
and one customer is permitted to depart (and proceed to the next queue if i < k). 
The number of departures fronl the i th queue up to time n is given by dl 1) (n). 
The second series of queues has t(D "moving" the custonlers in place of x. This 
time there are k - 1 queues. Initially, queues 2 thru k - 1 are enlpty and the first 
queue has infinitely many customers. There is a service event at the i th queue 
whenever t ) increases by one--that is, whenever, in the first series, there is a 
departure from the i th queue or an unused service at the (i + 1) th queue (these 
events will never occur simultaneously). 
The second series generates a new sequence of t's, which we denote by t (2) , and 
this is used to drive the third series, which consists of k - 2 queues, and so on. 
It is useful to define 
= _ z(J). 
(27) ql j) dl j) 
qJ) (n) is just the number of customers in the (i+ 1) th queue of the jth series at time 
n. For example, in the netwrk show in Figure 1, qï) = 0 and ql) = qï) = 1. 
Now consider the evolution of the corresponding semistandard tableaux T(n), 
n >_ 1. See Figure 2. Another look at the algorithm described above should 



A PATH-TRANSFORMATION FOR RANDOM WALKS 3677 

1 1 
2 2 
3 3 

I 
l 
d{31} I 

1 1 1 2 2 
2 13 3 3 
I I 
I I 
I I 
2 I q ql 
l 
I 
I 

3 3 

d(2 2) ,' 

FIGURE 2. The tableau T(17) 

convince the reader that T(rt) as precisely the semistandard tableau obtained when 
one applies the Robinson-Schensted algorithm, with column insertion, to the word 
al  "  an. 
To see this, look at the tableau T(17) represented in Figure 2. Recall that ara -- i 
if x increases by one at time m, in which case there as a service at the i th queue of 
the first series. 
Suppose that the next "letter" a(18) = 2. Sance d(21) < dï ), that as, ql) > 0, v 
have a departure from the second queue in the first series; that as, we decrease ql) 
by one and increase d( ) by one. In turn, this leads to an increase in t( ), that as, a 
service at the second queue in the second series, and so, sance qï) > 0, we decrease 
qï) by one, and increase d( 2) by one. That's it; the resulting tableau T(18) as 
1 1 1 1 1 2 2 3-- 
2 2 2 2 3 3 3 

and we recognise this procedure as column-insertion of the number 2 into the 
tableau T(17). 
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Note that., now, qï) = 0. Suppose that the next letter a(19) is also a 2. We 
(1) 
still have qï) > 0; so we decrease ql by one and increase d(2 ) by one. In turn, 
this leads to an increase in t(21), that is, a service at the second queue in the second 
series; but qï) = 0, and so this service is unused and there is no departure (that 
,(2) 
is, no increase in 2 )- Thc mmsed service leads to an increase in t 3), that is, 
a service at the only queue in t.he third series, and we increase d 3) by one. The 
resulting tableau r(19) is 
1 1 1 1 1 2 2 3 
2 2 2 2 2 3 3 

and, again, we recognise this procedure a.s cohmm-insertion of the nulnber 2 into 
the tableau r(18), and so o11. [] 
We will give a completely worked example, starting from an empty tableau, in 
the next section. 
We conclude this section with sonle relnarks oi1 the ilnmediate implications 
of Thcorem 3.1. Let l(n) and et(n) respectively denote the shape and weight 
of' T(lt). In view of T|leoreln 3.1. Lenlllla 2.4 can now be interpreted as stating 
that l(n) is the length of the longest non-decreasing subsequence in the reversed 
word an...al. This is a well-known property of the Robinson-Schensted algorithm. 
Thus, Lenmm 2.4 can be regarded as a corollary of Theorem 3.1, or the proof of 
Lemma 2.4 given in the appendix can be regarded as a new proof of the longest 
increasing subsequence property of the lobinson-Schensted algorithm. 
More generally, we can compare the statement of Theoreln 3.1 with Greene's 
theorem, and this leads to SOlne remarkable identifies. Greene's theorem (see, for 
example, [29]) states that, if m(n) denotes the llaximum of the Slllll of the lengths 
of i disjoint, non-decreasing subsequences in the reversed word a,...al, then, for 
(28) mi(n) =/l(n) +/2(n) +---+ li(n). 
It therefore follows from Theorem a.1 that 
(}) 
(29) mi(n) = Gk)(?i) ÷ G{k_)l(Tt ) ÷--" - Gk_i+l(Tt). 
It would be interesting to see a direct proof of this identity. Similarly, one can 
compare Theorem 3.1 with the various extensions of Greene's theorem given, for 
example, in [6]. 
The implications of Lemlna 2.2 for the Robinson-Schensted algorithm would 
appear to be less well known. Fix k _> 2, and set H(z) = F(e)(z*) *. 
Corollary 3.2. Suppose that (c,) holds. The weight e(n) of r(n) can be recovered 
from the sequence of shapes 
{sh (),  >_ ,}. 
In fact, if we set, for m >_ O, 
u(m) = sh r(n + m) - sh r(n), 
then 
a(rt) = sh r(n) + 
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Recall that the recording tableatoE er(n) are nested; the limiting standard tableau 
er(oo) = lim,_+ er(n) is thus a well-defined object. Il follows from Corollary a.2 
that provided (c,) holds we tan recover the infinite word ala2.., from er(oo). 
This is also tnle for the 1Robinson-Schensted algorithm applied with row inser- 
tion. To see this, recall that the recording tableaux maintained when one applies 
the row insertion algorithln to the infinite word alau.., is the smne as that lnain- 
tained when one applies the colunln insertiou algorithm to the word I 1 
ala 2 ... where 
a* = k - an + 1 (or see, for exalnple, [18, A.2, Exercise 5]). 

4. A WORKED EXAMPLE 

Suppose k = 3 and n = 7, and we apply the Robinson-Schensted algorithnl with 
cohlmn insertion to the word al  "a7 = 3112322. We obtain the following sequence 

of senlistandard tableaux: 

The evohltion of the corresponding queueing network is as follows. For each n, set 
Q(Tt) = [ q}l)(Tt) ql)(r/) 
qï() ] 

and 

Initially, 

D() = 

dï} (t)dl} (rt) d(31) (rt) 
d(3 
1 (-) 

Q(°)= [ 0 0 0] and D(0)= 

0 0 0 
0 0 
0 

Al lime 1, al = 3; so there is a service al the third queue in the first series. Since 
q) (0) = 0, this queue is empty, and the service is tmused, leading to an increase 
in t 1) and hence a service al the second queue in the second series. This is also 
unused; so we have a service al the first (and only) queue in the third series, and 
there is a departure there: d{3)(1) = 1. Thus, 

and D(1)= 

0 0 0 
0 0 
1 

The corresponding tableau r(1) is 
Al lime 2, au = 1; so there is a service al the first queue in the first series and 
a customer departs to join the second qlleUe: thus, dp)(2) = 1 and qï)(2) = 1. In 
the second series, this leads to an increase in t 1) and hence a departure from the 
first queue to the second quelle: thus, dï)(2) = 1 and qï)(2) = 1. In turn, this 
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leads to an increase in t 2) and hence a service at the oidy queue in the third series, 
which yields a (second) departure from that queue and we have d3)(2) = 2. Thus, 
1 0 0 
Q(2)= [ l 0] and D(2)= l 0 
1 2 

The corresponding tableau 7-(2) is 
Similarly, at time 3, a3 : 1, and we get 

and D(3) = 

2 0 0 
2 0 
3 

The corresponding tableau 7-(3) is 
1111131 
At time 4, aa = 2, and thcre is a departure from the second queue in the first 
series. This provides a service at the second queue in the second series, which is 
nonempty; so we have a departure from that queue as well. Thus, 

I2 1 0] 
Q(4)= [ 1 1 ] and D(4)= 2 1 . 
1 3 

The corresponding tableau 7-(4) is 
At rime 5, there is a service at the third queue in the first series, and a customer 
departs. That's all. Thus, 
2 1 1 
Q(5)= [ l o] and D(5)= 2 1 
1 3 

The corresponding tableau 7-(5) is 

1 3 
At time 6, there is a service at the second queue in the first series, and a customer 
departs; this yields a service at the second queue in the second series and a customer 
departs from there also. Thus, 
Q(6)= [0 1 ] 2 2 1 
0 and D(6)= 2 2 
3 

The corresponding tableau 7-(6) is 

21 1 3 
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At time 7, there is a service at the second queue in the first series, but this queue 
is empty and it is not used; this yields a service at the first queue in the second 
series, leading to a departure from that queue and consequently a service at (and 
departure from) the first (and only) queue in the third series. Thus, 

and D(7) = 

2 2 1 
3 2 
4 

The final tableau T(7) is 

5. IANDOM WALK IN A WEYL CHAMBER 
In this section we record some properties of the conditioned walk of Theorem 1.1, 
and extend its definition beyond the case p < -.- < pk- 
The random walk X is a Markov chain on Z+ with X(0) = o and transition 
matrix 
P(x, y) = py-Xl{y_xb}. 
Denote by P the law of the walk started at x E W ç Z+. 
We will refer to the random walk with p = .... pk as the homogeneous walk. 
Denote by st the Schur polynomial associated with the integer partition 
12 _> -..Ik _> 0. 
Lemma 5.1. For any r  7), the function hr " Z - , defined by 
hr(x) =p-s.(r)lxw, 
is harmonic for P. Note that h is strictly positive on W 
Proof. This follows immediately from the identity 
E st+«,(r) = st(r), 

i 
which in turn can be seen as a special case of the Weyl character formula, or can 
be verified directly using the formula 
 -'/dot( ). 
s(p) = det çp ) - 

Lemma 5.2. Suppose 0 < p < ... < pa. Then, for any x  V  Z_î_, 
P(X(n)  W, for ail n >_ O) = Cp-s«(p), 
where C is a constant independent of x. In particular, the transition matrix asso- 
ciated with the conditioned walk of Theorem 1.1 is given by 
(30) D(x,y) - p-ysy.(p) P(x,y) - sy.(p) 
p-s.(p) s.(p) l{Y-eb}" 
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Proof. When r = p, the Doob transform of the random walk X via the harmonic 
fimction h bas transition matrix/5. Note that this can also be regarded as the Doob 
transform of the homogeneous walk via the fimction x  sx. (kp). It follows from 
the asymptotic analysis of the Green function associated with the (Poissonized) 
homogeneous walk presented in [28] that, if n(x, y) is the Martin kernel associated 
with the homogeneous walk, then 
n(x,y)  constant x sx.(kp) 
whenever y tends to infinity in 1 in the direction p. Thus, by standard Doob- 
Hmt theory (see, for example, [15], [43]), any realisation of the corresponding Doob 
transform, sta.rting from the origin o, almost surely goes to infinity in the direction 
p. Moreover, any Doob transform on W which almost surely goes to infinity in the 
direction p is necessarily the smne Doob transform. It therefore suffices to show 
that the "properly" conditioned walk of Theorem 1.1, which is the Doob transform 
of X via the harmonic function 
g(x) = P«(X(n)  11 for ail n  0), 
almost surely goes to infinity in the direction p. But this follows immediately from 
the estinm.te, denoting the la.w of the conditioned walk by 
P(IX(n)-pnl > e,,)  P(IX(n)-p,l > en)/g(x) 
where c(e) > O. and a standard Borel-Cantelli argument. 
Note that the transition matrix  is well-defined by (30) for any p G P and. by 
the symmetry of the Schur polynomials, is symmetric in the pi. 
The proof of Lemma 5.2 given above is presented in more detail in [34], where 
an explicit formula for the constant C is also given. 

6. THE I::OBINSON-SCHENSTED ALGORITHM WITH RANDOM WORDS 
Having ruade the connection between the path-transformation G (k) and t.he 
Robinson-Schensted algorithm, we will now give a direct proof of Theorem 1.1. 
purely in the latter context. In fa.ct, we will present a more general result, which 
does not require the condition pi < "  < Pk- 
Let çl, 2,  -  be a sequence of independent random variables with common distri- 
bution p E 79. Let (S(n), T(n)) be the pair of selnistandard and standard tableaux 
associated, by the Robinson-Schensted correspondence (with column-insertion) 
with the ra.ndom word lç2""n. Here, T(n) is the recording tableau. Denote 
the shape of S(n) by ,k(n); the weight (or type) of S(n) is X(n), where 
X(n) --I{1 < m < n" . = i}l- 
Note that X is the random walk discussed throughout this paper, with transition 
matrix 
P(x, y) = pV-Xl{v_xeb}. 
The joint law of (S(n),T(n)) is given, for shcr = sh- - n, by 
(31) P(s(.) = 
and, forxEC={xGZ_" x >-.->x}, 
P(() = 
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Here p° is shorthand for pa, where a is the weight of the tableau er, and fx is the 
number of standard tableaux with shape x. The formula (31) follows immediately 
from the fact that the Robinson-Schensted correspondence with column-insertion, 
as in the case with row-insertion, is bijective. 
Cnsider the Doob transform of P on C, defined by its transition matrix 
p-ys(p) P(x, s(P) 
(32) Q(x,y) - p-Zsz(p) Y)- sz(p) l{u-zeb}" 
Theorem 6.1. A is a Markov chain on C with transition matrix Q. 
Proof. For x, y E C, we will write x ff' y if y - x E b. Recall that a standard tableau 
"r with entries { 1, 2,..., n} can be identified with a sequence of integer partitions 
/(i) ff'/(2) /--- ff" l(n), 
where l(m) is the shape of the subtableau of T consisting only of the eutrics 
{1, 2,..., ml. Since T(n) is a recording tableau, it is identified in this way with the 
sequence 
(1) ff" (2) ff" ... ff" (n). 
Thus, SUmlning (31) over semistandard tableaux er with a given shape l(n) l- n, we 

obtain 
(33) 

P(A(1) =/(1),..., A() = i(u)) = 

and so, for x ff" y l- n + 1, 

E p°=st(,)(p), 

sh cr=/(n) 

P(A(+I)=ylA(1)=/(1),...,A(-I)=/(n--1),A(n)=x) 

P(A(1) =/(1),...,A(n- 1) = l(n- 1),A(n) = x,A(n + 1) = y) 

P(A(1) =/(1) .... ,A(n- 1) = l(n- 1),A(n) = x) 

as required. [] 
Recalling the connection between G (k) and the Robinson-Schensted algorithm 
^ 
described in the previous section, and comparing Q with t.he transition matrix P 
defined by (30), we deduce the folowiug generalisation of Theorem 1.1. 
Corollary 6.2. Â" : G (k) (X) is a Markov chain on V Z with tradition matrix 
P. 

We will now record two lemmas which will yield an explicit description of the 
joint law of X and )ç, and an intertwining relationship between their respective 
transition matrices. 
Denote by ny the number of tableaux of shape x and weight y. (These are the 
Kostka numbers.) 

Lemma 6.3. 

(34) 
where 
(3) 

pY 
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Proof. First note that the a-algebra generated by {A(m), m _< n} is precisely the 
saine as the a-algebra generated by T(n). Thus, the conditional law of X(n), given 
{A(m), m _< n}, is the saine as the conditional law of X(n), given T(n). But this 
only depends on the shape, A(n), of T(n), and is given by 
pU 
'(:,V) := P(X() = ri () = :) -- 

as required. [] 

çVe will now show that P and Q are intertwined via the Markov kernel K: that 
is, 

Lemma 6.4. QK = IçP. 

Proof. We have 

On the other hand, 

(QI,)(:,z) = _, 
yEC 
p-Usu(p ) pZ 
=  P(x')p-s(p) su(p) 
yEC' 

(/,P)(x,z) =  
y 

Corollary 6.5. If we set J(x, y) = K(x*, y), then 
(36) P(X(n) = y I (m), m <_ n, (n) = x) = J(x,y) 

and J = JP. 
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For a discussion on the role of intertwining in the context of Pitman's 2/I - X 
theorem (the case k = 2), see [38]. 
It is instructive to note that Theorem 6.1 also follows from Lemmas 6.3 and 6.4, 
provided we can show that there is a class of functions of the form Kg, 9 " Z_ -- , 
which separate probability distributions on C. Indeed, by Lemm 6.3 and 6.4, 
E[(Kç)(A(n + 1)) I A(m),m  ni 
= E[E[ç(X(n+ 1))](n+ 1)]l(m),m  n] 
= E[E[ç(X(n+ 1))lA(m),m n+ 1],A(m),m n] 
= E[ç(X(n+ 1))lA(m),m  n] 
= K(A(n),y)E[ç(X(n+ 1))IX(n) = y] 
y 
=  a-(a(n),)  P(,z)(z) 
y z 
= [(a-P)](()) 
= [(ça)](()) 
= [ç()](()), 
which would imply that A is a Markov chain with transition matrix Q if the functions 
Kç were determining. To find such a cls of functions, we recall that the matrix 
{, (x,) e c } 
is invertible (see, for example, [31]). Thus, if we set, for q  N, 
zC 
we have (Kçq)(X) = q, and these fimctions are clearly determining. 
By extly the same arguments  those given in the proof of Theorem 6.1, if 
(n) denotes the shape of the tableau obtained by applying the Robinson-Schensted 
algorithm with row insertion to t.he random word ç -.-ç, we obtain 
Theorem 6.6.  is a Markov chain on C with transition matx Q. 

7. POISSONIZED VERSION 
We will now define a continuous version of G (k), and state Poissonized versions 
of Corollaries 6.2 and 6.5. This is an interesting setting in its own right, since 
the conditioned walk in this case is closely related to the Charlier ensemble and 
process (see, for example, [27], [28]), but more importantly it provides a convenient 
framework in which to apply Donsker's theorem and obtain the Brownian analogue 
of Corollary 6.2, as was presented in [36] in the case p = (l/k,..., l/k). Moreover, 
given the connection we have now made with the Robinson-Schensted algorithm, 
this continuous path-transformation can also be regarded as a continuous analogue 
of the Robinson-Schensted algorithm (see section 10 for further remarks in this 
direction). 
Let D0(l+) denote the space of cadlag paths f : I+ -- I with f(0) = 0. We 
will extend the definition of the operations / and v to a continuous context. For 
f, g E D0(l+), define f/ g E D0(l+) and f v g  D0(l+) by 
(38) (f/ g)(t) = inf If(s) + g(t) - g(s)] 
O<s<t 
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and 
(39) (fvg)(t) = sup If(s) + g(t)- g(s)]. 
O<s<t 
As in the discrete case, these operations are hot associative: un]ess otherwise de- 
lineated by parentheses, t.he default order of operations is from left to right: for 
example, whcn we write f A g A b, we mean (f A g) A h. 
Define a sequence of mappings F ()  Do(Ii+) k ---, D0(II+) k by 
(40) F(2)(f, g) = (f .q, gv f), 
and. for k > 2, 
(41) 
Let N = (NL .... , Ara) be a continuous-time random walk with generator Gf(x) 
= -i pi[f(x+ei)-f(x)]. Dcnotc bv lt{x thc law of N startcd from x, and by (Rt) the 
corrcsponding scmigrouI). ["or convenicnce, we will also dcnote the corresponding 
transition kerncl by Rt(x,y). Set p = tti/ll, and denote by = the law of the 
]tp-transforn on I', startcd at x, md by St thc corrcsponding senfigroup. 
Note that the embcddcd discrete-tilne random walk in N bas the saine law as 
X. That is, if Tri = inf{t _> 0" IN(t)l = n} and l'(,t) = N(Tn), then I" is a random 
walk on Z+ with transition llmrix P. 
Theorem 7.1. The law of/[ = F(a)(N) under 1o is the saine as the law of N 
under o. 
Moreover, if J is the Markov kernel defined in the previous section, then 
Theorem 7.2. 
(42) lRo(]V(t) = y I ]il(s), s <_ t) = J(M(t),y), 
and for t >_ 0 we bave StJ = JRt. 

8. BROWNIAN MOTION IN A WEYL CHAMBER AND RANDOM MATRICES 
Let X be a standard Brownian lnotion in N, and let lP= denote the law of \ 
started at x. Dcnote the corresponding semigroup and transition kernel by (Pi), 
and the natural filtration of X by 
Recall that 
and denote by/St the semigroup of the process killed at the first exit time 
T=inf{t>_0" X(t)IV}. 
Define Qx, for x G W °, by 
h(X(t / T)) 
where h is the Vandermonde function h(x) = Hi<j(xj - :ri). Denote the corre- 
sponding semigroup by 
The measure 
Qo = lira Qx 
II'o x---,O 
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is well-defined, and can be interpreted as the law of the eigenvalue-process associ- 
ated with Hermitian Brownian ,notion [17], [2o]. The lw of X(1) mdr Qo is the 
familiar Gaussian Unitary Ensemble (GUE) of random matrix theory. 
In [36] it was shown, by applying Donsker's theorem in the context of Theo- 
rem ï.1 with p = (1 .... ,1), that 

Theorem 8.1. The law of F(d)(x) under Po is the same as the law of X under 
Qo. 

In particular, 1 (Xdv -'- v X1)(1) llas the saine law as the largest eigenvalue of 
a d x d GUE randoln nlatrix; this had been observed earlier by Baryshnikov [5] and 
by Grawmr, Tracy and Widom [21]. A similar representation was obtained in [11]. 
Here we record some additional propertiis of the process R = v(d)(x), and 
its relationship with X, which are inherited, in the sanie application of Donsker's 
theorem, from Theorem 7.2. 

Theorem 8.2. 
(43) o(X(t) 
where L is characterised by 
 (let(e,) 
(44)  eyL(x'dy) = h(X)h(x) =: c(x). 
Also, for t  0 we bave QtL = LPt. 
The intertwining QtL = LPt can also be Seell as a direct consequence of the 
Harish-Chandra/Itzykson-Zuber fornmla [22], [25] for the Laplace transform of the 
conditional law of the diagonal of a GUE random matrix given its eigenvalues, using 
the fact that the diagonal of a Hcrmitian Brownian motion evolves according to the 
oemigroup Pt and the eigenvalues evolve according to the Senligroup Qt- It is also 
easily verified, using the Karlin-MacGregor formula, that 
t(*,Y) =  sgI,(a)Pt(x, ag). 
We will now present analogous results for Brownian motion with drift. Fix 
p  a, and denote by FP) the law of Brownian motion in a with drifl p. Denote 
by (PP)) the corresponding semigroup and by (P)) the semigroup of the process 
killed at the first exit timc T of the yl chamber IV. Define bu by 
(45) bux ) = e -' det (e u'). 
It is easy to check directly that b. u is a positive harmonic function for u). Define 
Denote the corresponding semigroup by QP). Recalling the absolute continuitv 
relationship 

(46) 

ff(t ") (x, y) = et"(-)-IltqV==t/2(x, y), 

ISee Remark 2(ii) in section 10 below 
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we can write 
det (e.X) 
and we note that this is symmetric in the p. 
It is ey to verify that the mesure 

() - lira Q() 
W O x--*O 

(5, y), 

is well-defined. 
Applying Donsker's theorem in the coutext of Theorem 7.1, as in [36], we obtain 
Theorern 8.3. The law ofF(d)(x) under ]P(o ) is the same as the law of X under 
As in the discrete case, we remark that the law (t,) is symmetric in the drifts 
Iii. 
We also have, by the saine application of Donsker's theorem, the following ana- 
logue of Theorem 7.2. 
Theorem 8.4. 
(48) )(X(t)  dz I R(s), s  t;R(t)=r)= L()(r, dx), 
whe 
(49) L¢")(,@) = «.()-"L(, @). 
Also, for t  O, 
(50) Q') L (') = L(')P 
The intertwining relationship (50) can also be verified directly using QL 
and (46). 
For related work on reflecting Brownian motions and non-colliding diffusions 
see [7], [11], [12], [13], [16], [23], [35], [39] and references therein. 
9. AN APPLICATION IN QUEUEING THEORY 
In this section, using the connection with the Robinson-Schensted correspon- 
dence obtained in section 3, we will write down a formula for the "transient distri- 
bution" of a series of 51/51/1 queues in tandem. There are many papers on this 
topic for the ce of a single queue, where the solution is given in terres of modified 
Bessel functions; see [2] and references therein. In [3], the ce of two queues w 
considered and a solution obtained, but the techniques used there do hot seem to 
extend eily to higher dimensions. 
Consider a series of MIMI1 queues in tandem, k in number, driven by Poisson 
processes N,...,Nk with respective intensities , .... pk. The first queue h 
infinitely many customers, and the remaining queues are initially empty. At every 
point of Ni, there is a service at the i h queue and, provided that queue is hOt 
empty, a customer departs and joins the (i + 1) h queue (or leaves the system if 
i = k). 
Denote by D(t) = (D(t),...,D(t)) the respective numbers of customers to 
depart from each queue up to time t. Note that, since there are always infinitely 
many customers in the first queue, D is a Poisson process; we can thus ignore the 
first queue, think of the second queue  the first in a series of k - 1 queues, and 



A PATH-TRANSFORMATION FOR RANDOM WALKS 3689 

think of D1 as the arrival process at the first of these k - 1 queues in the series. 
This is a more conventional setup in queueing theory. The state of the system is 
described by the queue lengths 

(51) 

Q1 = D1- D2,..., Qk-1 = Dk-1- Dk. 

We will write down a formula for the law of D(t) which, in turn, yields the law of 
Q(t) -= (Ql(t),...,Qk-l(t)). 
Without loss of generality, we can assume that IPl = l. The de-Poissonized 
version of this problem is to consider the usual random walk X, with p = p, and 
consider the law of 5(n) = (D(k)(X))(n). But we know this law, from sections 3 
and 6. It is the law of/3(S(n)), where S(n) of the random semistandard tableau 
obtained when one applies the Robinson-Schensted algorithm with column-insertion 
to the random word 1 ---k and fli('r) denotes the number of i's in the i th row of 

a tableau "r. 
Thus, 

(52) P(D(t) = d) = e -t Z P(5() = d), 
n>_O 

where 
(53) 

P((n)=d)= E E P'-fl{N'-)=d}" 
l>_d,lt-n sh 
This formula is COluplicated in general, but simplifies lu certain cases. 
Consider the case k = 2. In this case, we have only one summand: 
(54) P(5(n) d) = a,n-alr 
: P1 P2 (n--d2,d2)" 
By the hook-lenh formula (see, for example, [18]), for n 
n- 2d2 + 1 
(n- d2 + 1)[d2]" 

f(n-d2,d2) = n! 
Thus, recalling that p = p, 
(55) P(D(t) = d) = e - 

It follows that 

Z n dlt-dl 
t Pi 
n>_dl +de 

n - 2d2 + 1 

(n- d2 + 1)!d2!" 

(56) P(Q(t) =q) = (p/p)qe -t  (m + 1)(pt)'Im+(2 PV/-fiït). 
m>_q 
Let sud denote the Schur polynonial associated with the skew-tableau 1/d (see 
for example, [18]). We will use the following formula: 
(57) E sud(x)tlt'f t)dl fd 
  - ' ICI! " 
This follows from the identity 
(58)  sud(x)s,(y) = sa(y) (1 - ij) -1 
l ij 
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(this is a variant of Cauchy's identity; see, for example, [31, pp. 62-70]) and the 
fact that 
liln s, ( t-.ln' 
(59) 
In the case k = 3, i p2 = p3, we bave 
p3) tlll ft 
P(D(t) = d, D  d, Dz(t) = dz) = 
(60) -p, t. d tldl fd 
It would be interesting to compare (61)) with the explicit fornmlas obtained in [3] 
for this case. 
In the geueral case, we can simplify the formula (52) if d is constant. Suppose 
di = m for all i. Thon, using (57) and thc hook-length formula, 
I1 

(62) 

(63) 

It follows that 

P(D(t)=d) = e-tpaZst/a(p2 
I 

= e-»l'(tb'w) TM 1-I r(i + ,,) 
i<_a F(i) 

(64) 
where 

P(Qt (t) = Q2(t) .... = Q]¢-I (t) = O) ---- e -plt tt(tkplp2 " " " ph-), 

r() 
(65) () =  « 1-I r(i + ,) 
rrt  O 

Finally we remark that. by Theorem 6.1 and the symmetry of the Schur poly- 
nomials, the law of the process D is svnnnetric in the parameters pi,... 

10. CONCLUDING RElklARKS 
1. The h rawchouk process: A binomial version of Theorem 1.1 was presented in 
[28]. This states that, if X is a random walk in Z_î_ with transition matrix 
P(x, y) = 
(C is a normalising constant), then, assuming p < -.- < Pk and extending slightly 
the domain of G (), we see that G(k)(X) bas the saine law as that of X conditioned 
to stay forever in II'. In this case, a similar connection can be ruade with the 
dual 1Robinson-Schensted-Knuth (1RSK) correspondence for zero-one matrices, and 
analogues of all the main results of section 6 can be obtained similarly. The Schur 
polynomials again play an important role. See [34] for details. 
2. Properties of F(): The following continuous analogues of Lemmas 2.1-2.4 
can be readily verified. Denote by C0(+, ) the set of continuous functions 
f "IR+ ---, IR ¢ with f(0) = o. For f  C0(IR+, IR¢), where k _> 2, 
(i) IF()(f)I = 
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(ii) F(k)(f) = fk v"" v fl, 
(iii) f(t) = [F()(f)](t) + ()([F()(f)](t, u), u _> t), where () is defined on a 
suitable domain as the COltiliuOUS almlogue of F (k). 
In this continuous setting, the identity (ii) can be verified directly using the "sup- 

integration by parts" forlmfla 
0<s<t 1.0<r<s 0<s<t 

sup v(r) } 
0<r<s 
= sui)u(s)+ sup v(s), 
O<s<t 0<s<t 

for u, v E C0(1R+, 1R). This is a "max-plns" mmlogue of the usual integration by 
parts fornlula and is easily verified by the method of Laplace. 
3. A continuous Robinson-Schensted algorithm: Given the colmeclion we have 
ruade between the path-tralsforllmtions (7 (1 and the lqo|finson-S«hensted algo- 
rithln, the mappings F () tan be used to define a continuous version of the l:{obinson- 
Schensted algorithln. More precisely, let Çc denote the Gelfand-Cthn cone, 
whiçh consists of triangular arrays of l'cal nUln|mrs 
(66) 
txj , l<i<k, l<_j<_i) 
satisfying xj-(i) _> xj-(i-1) _> "J+'Ai) for all i,j. Points in the Gelfand-Cetlin cone can 
be regarded as continuous analogues of selnistaldard tableaux. The continuous 
analogue of a word is a continuous fimction f " [0, 1] -- ]R k wilh f(0) = o: denote 
the set of these functions by C0([0.1],]Rk). Define a lnap ¢" C0([0, 1],]R 
as follows. For conveuience, let F (1) be the identity tralsformation. If we set 
x = ¢(f,...,fk), then, for each 1 < i < k, 
(67) x (i) = ([Fl i) (Il ..... ri)](1) .... , [F i) (f .... , f,)](1)). 
The continuous analogue of the corresponding "recording tableau" is the path 
(68) p(f) = {[r{)(f)](t), 0 < t < 1} E C0([0, 1], |l'). 
By analogy with the discrete IRobinson-Schensted algorithln, the function f can be 
uniquely recovered from the pair ¢(f) and p(f). A more detailed discussion on 
the properties of this continuous Robinson-Schensted algorithm will be presented 
elsewhere. 
4. GUE minors: Let A be a k x k GUE random matrix, and denote the eigen- 
values of the i th lllillor (Azm, 1, rn _< i) bv. A i) -> "'" -> -.i)'(i), for i _< k. Iii the above 
context, Baryshnikov [5] showed that, if (B(t), 0 < t < 1) is a standard Brownian 
motion in 1R , then the randoln vector 

has the saine lave as 

(¢')(B),..., Cï)(B)) 

In [5], Donsker's theorenl is applied in the context of a raildom senlistandard tableau 
with the same law as T(n) of section 6 in the homogeneous case Pi = .... p. We 
can thus extend Baryshnikov's arguments using the representation for the Robinson- 
Schensted algorithm given in this paper and the continuity of the mappings F (i), 
i _< k, to sec that, in fact, ¢(B) has the saine law as 

(A () .... , A()). 
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However, it is easy to see, 2 by considering the case k = 2, that these identities do 
hot extend to the process level (that is, with ¢ defined simultaneously on intervals 
[0,t] instead of just [0, 1] and "GUE" replaced by "Hermitian Brownian motion"). 
5. Related topics: The intertwining of Lemlna 6.4 is closely related to the work of 
Biane on quantmn random walks [8], [9], [10]. The Robinson-Schensted correspon- 
dence is of fundamental importance to the representation theory of S, and GL(n). 
Related topics in representation theory (which are certainly connected to results 
presented in this paper) include Littelmann's path model for the finite-dimensional 
representations of GL(n) (see, for example, [30]), crystal bases, and representations 
of quantm, groups (see, for example, [14], [29]). 

APPENDIX 
Proof of Lemma 2.1. The first identity is trivial: 
(yvx)() = max [y(.) + x() - (')l 
0<m<n 
= () + () + ma [(.) - () - æ(-)l 
0<m<n 
= () + (,) - ( , )(). 
If we set z = y - x, and s(n) = max0<m<n z(n), then the second identit¥ is 
equivalent to the well-known fact that 

s(n) = min[2s(/)- z(/)l. 
l<n 

Proof of Lemma 2.2. Fix x E Ak, and write G (k) = G()(x) unless otherwise indi- 
cated (similarly for D () and T()). We will first show that [G()[ = Ix[. We will 
prove this by induction on k. The case k = 2 is given by Lenmm 2.1. Assume the 
induction hypothesis for k - 1. We recall from the definitions that 
By the induction hypothesis, 

(70) 
Recall that, for i > 2, 
= D () 
(71) 
and 
(72) T(_ï 
Thus, by Lemma 2. 
(73) x = D k) r() 

for each i > 2; summing this over i and recalling that Dï ) 
(7) Ixl = D( ) + 
so we are done. 

= xl yields 

2Bougerol and Jeulin, private communication 
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We will now show that 
(75) x(n) = c(k)(.) + F () (C()(t) - C()(n), t >_ .), 
for some function F (k) to be defined. Again we will prove this by induction on k, 
and note that for k = 2, this is given by Lemma 2.1. 
A recursive definition of F (k) will be ilnplicit in the induction argument. Recall 
that 
(7) 
Assuming the induction hypothesis for k - 1, we have, for i  2, 
(77) 
Thé, for i  2, using (73) and the ft that 
D)(n) - 

we have 

(k) () 
where J[) is defined on a suitable domain. It is important to note here that J[) 
does not depend on n. 
In this way, recalling that D ) = G ), we obtain 
where the function (k) is implicitly defined by this identity (on a suitable domain) 
d does not depend on n. Observe that we can also recover the sequence of future 
increments xk (1)- xk (n)  a function, which does not depend on n, of the sequence 
{c()(/) _ c()(),   }. 
We can now recover the values Xk-l(n), Xk-2(n), and so on,  follows. By 
equations (73) with i = k, 
= _ T(k) 
(s0) Dçi() )() () +--i()- 
It follows that the sequence {Dl(n,l), l  n} is a function, which does not 
depend on n, of the sequence {G(k)(/) -G(k)(n), l  n}. Combining this with (78), 
we see that 
(81) Xk-l(n) = Gl(n) + Fî)l 
where )1 is implicitly defined by this identity (on a suitable domain) and does 
not depend on n. Similarly, we can recover the sequence of future increments 
xk(1) -xk (n)  a function, which does not depend on n, of the sequence (G (k) (1) - 
G()(n), l  n}, and so on. Finally, xi(n) is obtained using Ix] 
Proof of Lemma 2.3. We want to show that, for (a, b, c)  A3, 
(82) a  (c  b)  (b c) = a  b c, 
and for (w, x, y)  
(83) 
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First note that these identities are equivalent. To sec this, set a(n) = n - w(n), 
b(,,) = n- x(,,) and c() = ,t-y(n). Then I,lug these into (82) to obtain (83). 
will therefore restrict our attention to the identity (83). 
Let d=xy,t=yvx, q=x-dand'u=y-d. Then (83) becomes 
(84) ,( + )  (- ) =xy. 
That is, the output of a series of queues in tandem driven bv (w, x + u, y - u) is 
thc saine  that of the series driven by (w, x, y). Set 
d 1 = f  ' 

and 

ql = w -- dl, 
q2 : dl -- d2, 
O = L,- ,, 
42 = ïl-J» 
We want t.o show that d 2 ----- Ï2. From the above definitions, this is equivalent to 
showing that 
q,(,) + q() = 0() + 0() 
for ail n  0. We will prove this by induction on . 
The induction hypothesis H is: 
q + q2 = 01 + 2, and either 
(i) 2 -- q2 OE 0 and q - q2 = 0, or 
(ii) ç2 - q2 = 0 and q - q2  0. 
When  = 0 we bave q = ql = q2 = 1 = 2 = 0. and the induction hypothesis is 
trivially satisfied. Assmne the iuductiou hypothesis holds at rime n- 1. Note that 
(w, x, y - , u)  A4; that is. ouly one of these quantities, if any, can increase by 
one at time n. We will consider the following rive cases, which are exhaustive and 
mutually exclusive, separately. 
() (,,, v- , )()= (',,v-,, )(- ), 
(b) ()- ,(- 1)= 1, 
(«) *()- .(.- )= , 
(d) (V- )() - (V- )(- ) = , 
() (.)- (.- )= 1. 
Case (a): (w,x,y- u,u)() = (w,x,y - u,u)( - 1). In lhis case, nothing 
changes, and so H is preserved. 
Case (b): w()-w(n- 1) = 1. Iu this case, q() = q(n- 1) +1 and 
01 (n) = 0 (n- 1) + 1, the other quautities remain unchanged, and H is preserved. 
Ca» («): x()- z( - 1)= . Th V() =V( - 1)+ 1. 
Sppos ql( - ) > (- ) > 0. Th ql() = ql(- )-  nd q() = 
q2(n -- 1) + 1. Thus, q - q2 and ql + q2 do hot change. Also, O(n) = 1( - 1) - 1 
and 2() = 02( n - 1) + 1. Thus, 2 - q2 aud 0 + 2 do hot chauge eitherç so we 
still have ql + q2 = 01 + 2, and H is preserved. 
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Now suppose ql( -- 1) > 01(' - 1) = I). Note that this ilnplies ç2(n - 1) - 
q2( - 1) > 0, so that we are initially in case (i) of the induction hypothesis. In this 
case, q(n) = ql(n - 1) - 1 and q(,) = qe(- - 1) + 1, but 01 and ç do hot change. 
Thus, q - q, q + q2 and Ç1 + 02 do llOt change. Thc quantity 0 - q decreases bv 
one, but remains nolmegative; so we relnain in case (i), and H is preserved. 
Finally, if ql (n- 1) = 01 (l!- 1) = 0, then we arc initially in case (il) of H. There 
is no change to q, q, O or ç, but q inceases by one, and so we remain in case (ii), 
and H is preserved. 
Case (d): (V - u)(?t) - ( - .u)(?l- 1) = l. In this case, q(?t- 1) > 0 alld q 
decreases by one. The values of ql and Ol do hot change. 
If we are iii case (i) at rime n - 1, lllen 02(n - 1) k q2( - 1) > 0, and so both 
q2 and 02 also decrease bv one; thus, we remain iii case (i), and H is preserved. 
If we are in case (ii) at tinle t - 1, then ç2( - 1) = q2( - 1), and either q and 
02 both decrease by one or both rclnail undlanged. Either way, we remain in case 
(il), and H is preserved. 
Case (e): u(n)- u(,t- 1) = 1. Then q(n- 1) = q2(n- 1) : 0, al,d we are 
ilfitiallv in case (i) of H. The values of q, q and q will nol change. If çl > 0. then 
çl decreases by one and O2 increases by one; olherwise, Ol and O2 do hot change. 
Eithcr ww, we relnain in case (i), and H is prescrved. 
Proof of Lemma 2..  will prove lhis by induction on k. It is certainly true for 
k = 2, from the definition of G (). Recall the definition of G (k), 
(85) 
By the induction hypolhesis, that the lemlna is lrue for G (-n, we bave 
(86) c > <> <> Tï ) 
: k_11_2 "'"   
XOE will now rptly pply Lmnm 
Ta() () 
-1I-2 : xkD I 
(k) 
= k  2k-1  k-2" 

Similarly, 
,(k) 

= £ V Xk--1 V ,I'k--2 V D 

and so on. [] 
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ABSTRACT. For a prime IniInber p and a number fie|d k, let A denote the 
projective limit of the p-parts of the idea| class groups of the intermediate 
fields of the cyclotomic Zp-extension over k. It is conjectured that A is finite 
if k is totally reM. When p is an odd prime and k is a real abelian field, we give 
a criterion for the conjecture, which is a generalization of results of Ichimura 
and Sumida. Furthermore, in a special case where p divides the degree of k, 
we also obtain a rather simple criterion. 

1. INTRODUCTION 

Let p be an odd prine number. For a mmlber field k, let k denote the cy- 
clotomic Zv-extension of k with its 'n-th laver kn ( > 0). \Ve denote bv A the 
projective limit of the p-Sylow subgroup of the idem class group of each kn with 
respect to the relative norms. If k is a totally rem mnnber fiel& it is conjeçtured 
that A. is a finite abelian group ([7], [12, p. 316]), which is often called Greenberg's 
conjecture. 
When k is a real abelian field whose degree is hot divisible by p, Ichimura and 
Sumida ([8], [9], [10]) discovered a good Inethod for veri'ing the conjecture. In 
this case, we can decompose Ax into a direct snnl of its x-parts .4oe for Dirichlet 
characters X. corresponding to k. Then they gave a necessary and sufficieut condition 
for A to be finite in tenns of certain cyclotomic traits and some polynomials related 
to the Kubota-Leopoldt p-adic L-flmction Lv(s, \) associated to \. It is suitable 
for a practical computer calculation and, for exmnple, using it they showed that the 
conjecture holds when p = 3 and k = Q(v/-) with 1 < ni < 10 4. (Similar criteria 
are also obtained by [13] and [14].) 
In this paper, we study the case whcre k is a real abelian field of arbitrary degree, 
including the case p I [k : Q]. Although we cannot necessarily decompose .4 into 
direct summands by using characters \ in this case, the conjecture for k follows 
from the finiteness of all (suitably defined) k-parts 4 of 4¢ (cf. Lemma 2.1). 
The finiteness of A is also conjectured, and the purpose of this paper is to give 
some criteria for this conjecture for \ to be true. 
First, we will generalize the criterion of Ichimura and Smnida to arbitrary char- 
acters, especially characters of order divisible by p (Theoreln 2.6). In their proof, 
IchiInura and SuInida ([8], [10]) essentially used a theorem of Iwasawa [11] and 
Gillard [6] which describes the Galois module structure of "senti-local units modulo 
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cyclotomic units" 1Xx/C x in terres of the p-adic L-function Lp(s, X). Iwasawa and 
Gillard dealt with only the case where the order of \ is hot divisible by p, and this 
is a main reason why Ichimura and Sumida imposed the saine assumption. The au- 
thor [18] removed the assumption on the order of k in the theorem of Iwasawa and 
Gillard and determined the structure of 1Xx/C x for arbitrary  (Fact 2). By using 
this result, we can prove out theorem by the saine way as the proof of Ichimura and 
Sumida. When the order of t is p and -((p) = 1, Fukuda and Komatsu [5] obtained 
a similar criterion. 
In a special case where the order of t is divisible by p, we also obtain the following 
simple sufficient condition for the conjecture to be true: 
Theorem (Corollary 2.4). Assume that .Xv-(p) is a nontrivial p-power root of 
unity, where w is the Teichmiiller character. If the distinguished polynomial Pt (T) 
associated to the p-adic L-function Lp(s, \) is irreducible, then A is finite. 
The assumption in the above theorem is satisfied only when the order of \ is 
divisible by p, since the order of z is prime to p. In particular, this case was hot 
dealt with in the papers [8], [9] and [10] of I«himura and Sumida. We will prove 
the above theorem by using the fact proved in [18] that 1Xt/C t is hot necessarily a 
cyclic module. 
We remark that there indeed exist characters \ satis-ing the condition in the 
above theorem. We give some numerical examples in §6. We also veri .fy in §6 the 
conjecture for several real abelian fields by applying our theorem, a generalized 
criterion of Ichimura and Sumida. to sonne characters \. 

2. IAIN RESULTS 

\Ve fix an odd prime number p. Let \ be a nontrivial Qp-valued even Dirichlet 
character of the first kind, i.e., the conductor of y is not divisible by p2, and let 
k = kx be the fixed field of the kernel of \. We denote by k the cvclotomic Ep- 
extension of k with its n-th layer k. (n >_ 0). We write A, = Ak, for the p-Sylow 
subgroup of the ideal class group of k, and put .4 -= .4 := li.___m A,, the projective 
limit being taken with respect to the relative norms. We put A := Gal(k/Q) and 
F := Gal(k/k); so Gal(k/Q) =  × F, since \ is of the first kind. Then we 
regard As as a module over the completed group ring Ep[A]F. It is well known 
that .4= is finitely generated and torsion over Ep[A]F ([12, Theorem 5]). Let 49 
denote the ring generated by the values of \ over Zp. For any Ep[A]-nodule .I, 
we define an 49-module M t, the .k-part of M. by 

Mt := {m e M ®z, O ] 6m = (6)m V6 e 
(for the properties of the x-part, cf. e.g. [17, §II.1] and [18, §21). If 
conjugate of "¢. then .li t' - .li t. Then O-modules .4 are defined (0 _< 
and A = li___m A becomes an O[[F-module. Let f be the prime-to-p part of the 
conductor of \ and put q = fp. Denote by p, the group of pn-th roots of unity 
for n >_ 0 and/Zp¢ := (.J/Zp,. Identi-ing F with Gal(k(lp)/k(lv) ) in a natural 
woEv, we choose and fix the topological generator -)' of F such that Ç-r = Ç+q for 
all Ç ff/z. We identi', as usual, the completed group ring O[F] with the power 
series ring A := O[T] b." q = 1 + T. Thus, AX is regarded as a module over A, and 
is finitely generated and torsion over A. For a finitely generated torsion A-module 
3I. denote by charAM the characteristic polynomial of M, which is a uniquely 
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deterlnined distinguished polynomial times a power of a fixed prinle element of (9. 
We denote by Ax (resp. #<) the A-invariant (resp. #-invariant) of A. We know 
that #x = 0 by the theorem of Ferrero and Washington [4]. Greenberg's conjecture 
for \- is as follows: 
Conjecture. Let  be an even Dirichlet character of the first kind. It is conject.ured 
that A L is finite, that is, charAA = 1 or A x = O. 
Greenberg's conjecture for a real abelian field implies the above. Indeed, by 
using the theorem of Ferrero and Washington [4], one can show the following: 
Lemma 2.1. Let If be a reai abelian fieid. Then AI( is finite if and only if A = 0 
for ail representatives of Qp-conjugate classes of Dirichlet characters \ of the first 
kind with k< C K. 
In our main results of this paper, we give two criteria for the conjecture for 
To state these, we need to recall the relation between charAA and the Kubota- 
Leopoldt p-adic L-function Lp(s, () associated to ,(. By Iwasawa, there exists a 
unique power series gx (T) in (.gIT 1 such that 
g((1 +q)S_ 1)= Lp(1- 
(cf. [20, Theorem 7.10]). Using the p-adic Weierstrass preparation theorem raid the 
theorem of Ferrero and Washington [4], one can uniquely write 
(2.1) g<(T) = ux(T)Pn(T ) 
for a distinguished polynomial Px in O[T] and a unit u(T) of A. Put A = 
degP x (T). It follows from the Iwasawa main conjecture proved in [16] that 
(2.2) charA l Px(T), 
and hence A n _< A (see (3.2) and Fact 1 in §3). Therefore we bave charAoe = 1 if 
and only if P(T)  charA for ail distinguished irreducible factors P(T) of P(T). 
One of our main theorems of this paper is as follows: 
Definition. For a distinguished polynoufial P(T) in OIT] such that P(T) I Px (T), 
we define a condition (7-) as follows: 
(7") vo(Q(q)) < vo(1- (w-l(p)) where Q(T)= P,(T)/P(T), 
where vo denotes an additive valuation of 0 and w the Teichmiiller character. 
Then we have 
Theorem 2.2. Let P(T) be a distinguished polynomiai in O[T] such that P(T) 
Px(T). Assume that P(T) satisfies (T). Then we bave P(T) { charA. 
For the condition (7"), we can easily see the following: 
Lemma 2.3. 
(i) /f Xw-(p)  pp, there exists no factor of P(T) satisfying (7). In par- 
ticular, if the order of  is prime fo p and \w-l(p)  1. there exists no 
factor of Px (T) satisfying (7). 
(ii) If xw-l(p)  1, a factor P(T) of Px(T) satisfies (7) if and only if 
vo(P(q)) > 
hoids, where Bl,x,- denotes the first generalized Bernouili number. 
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(iii) If \c-(p) Ç pp, then Px(T) satifies (T). 
(iv) ([3, Proposition 2]) /.f \0.1--1(])) ---- 1. te, T-q divides Px(T) and s the 
miqu,e irreducible pol'ynomio.l satisfying (T). 
In the case where kw-l(p) = 1. by (iv) of the above lemma, Theorem 2.2 just 
a.sserts that 
T-qIPx(T ) and T-q{charAA L. 
This is already proved under the assumption that the order of k is prime to p ([9, 
Remark 5]). For other cases, we ol)tain the following. 
Coronary 2.4. A.s«.e tl«t -(p)  U  {1}. Tten P(T)  charna. 
Furth«rmor«, ,¢ P,(T) is irreducibl«, or if uO(l.kw- ) = 0. ,« bau« x = O. 
 will give in 6 s()me cxanll)les satisfying the conditions in the above corollary. 
Rroark 1. For a.n arbitrary cven Dirichlct çharacter  of the first kind. we can 
show that vo(B«-) = 0 implies ,4 = {1} for ail n  0 (see 5). 
Next. we stalc thc othcr lnain thcorcm of this pat)er, which is a generalization 
of thc rcsult of Ichinlura and Sumida [8], [10] to arbitrary characters. We have to 
prcpare some notation. We fix a (listinguished polynolnial P(T) in OIT] such that 
P(T) I Px(T). Put, .= (T) = (1 + T)p" -  and  = (T) =/T for n  0. 
Bv the Leopoldt. «oloe«ture for p and kn proved in [1], A/( P, wh) and A/(P, un) are 
finite abelian groups for any n  0. b denote by noe, the exponent of A/(P,w) 
(resp. A/(P, un)) if k(P)  1 (resp. k(P) = 1). Then we take a polynonfial Xoe.n(T) 
in O[T] satisfying 
(2.3) Xp,(T)P(T)  mp mod  if k(P) # 1, 
' ( u. if (p) = 1. 
This polynomial -Yoe, n is uniquely determine(l inodulo 'n (resp. un) since w and 
P(T) are relativelv prime. Choose an element , in Z[][T] such that 
where ç, is an elcment of Zv[][T] satis'ing  (À,) = X,. Here we regard 
   z[r]-ilr o.onorhil. z[l[rl  O[] i1«.¢ »- » ¢t . (r. 
) dcnote the Sylow subgroup (resp. thc prime-to-p part) of &: 
 = p x '. 

We put b = YI/x'. Let 

1 
« := #,  T(g'()) - 
6EA' 
be the idempotent of Zp[A'] corresponding to b, where Tr is the trace map from the 
field generated by the values of  over Qv to Qv. Let  Ç Z[] denote an element 
of p of order p (resp. . = 0) if p is nontrivial (resp. trivial), and ex = 
an elelnent of Z[] such that 
ex  e(1 - ) mod mv,. 
For any m  1, we fix a primitive -th root Çm of unitv with the property that 
,, = Ç for all m'  1.  use cyclotolnic elements in k defined as follows: 
c = N(¢,,,+, )/,, (1 - Q.+, )t-. 
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where t denotes the cardinality of the residue class field of a prime ideal of k over 
p. If f ¢ 1, i.e., the conductor of ,( is not equal to p, then the element ch is a 
(cyclotomic) unit of k,. Since the case where f = 1 is treated in [8], [9] and [10], 
we assume f  1 in all that follows. By the identification 3' = 1 + T, the element 
Yp,, of Z[A][T] can act on the group k x . For each n >_ 0 consider the following 
condition: 
We can see that the condition (Hp,,J does hot depend on the choices of Xp,, 
and Yp,. We remark that, in the case where ,(p) = 1, the condition (Hp, o) does 
hot hold since c0 = 1 and mp, o = 1 for any P(T). The following lemma can be 
proved in a way similar to [9. Lemma 1] by using Lemma 4.1. 
Lemma 2.5. The condition (Hp, n) implies (HP, n+l). 
Our main theorem is stated as follows: 
Theorem 2.6. Let P(T) be a distinÆuished polynomial in O[T] such that P(T) 
P,,(T). Then we bave P(T)  charAA if and only if the condition (Hp., 0 holds 
for some  >_ O. 
The above theorem coincides with theorems in [8] and [10] if the exponent of A 
divides p - 1. 
By Theorems 2.2, 2.6 and (2.2), we obtain 
Corollary 2.7. We bave A< = 0 if and only if for any distiguished irreducible 
polynomial P(T) such that P(T) I Px(T) and P(T) does hot satisfy (T), the con- 
dition (Hp,, 0 holds for some n >_ O. 
By the Chebotarev density theorem, we obtain the following: 
Corollary 2.8. We bave  =- 0 if and only if for any distinguished irreducible 
factor P(T) of P<(T) that does not satisfy (T), there exist an integer n OE 0 and a 
prime ideal [ of kn of degree one for which the condition 
«e xYP'" modl¢ 
holds. Here IZ = I ç Q. 

3. PRELIMINARIES 
We first recall the Iwasawa main conjecture and show its consequence (2.2). Let 
M/k be the maximal abelian p-extension unramified outside p, and let L/k be 
the maximal unramified abelian p-extension. As usual, we consider Gal(M/k), 
Gal(L/k) and Gal(M/L) as Z[A]I[r-modules. Let p be a prime ideal of k over 
p. There exists a unique prime ideal p, of k, over p since k is of the first kind. We 
denote by U,,v the group of principal units in the completion k,.v of k, at p,. Put 
where p runs over all prime ideals of k over p. Let E be the group of units e of 
k, such that e = 1 mod p, for all p, I P- Let , be the closure of the image of E 
under the diagonal map E - b/,. Put 
L/:= lim L/,,  := lira ,, 
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where the projective limits are taken with respect to the relative norms. We regard 
/A and L" as modules over xp[x]lr. By class field theory, we have the following 
isolnorphislns of Zp[A] IF]-inodules: 
(3.1) A  Gal(L/k), 1A/£ ,2_ Gai(M/L). 
(For the latter, see [9, Lemma 3].) Put 
: := Gal(M/k). 
It is known that X is finitely generated and torsion over Xp[A]F ([12, Theorem 17]), 
and we further see that  is finitely generated as a Zp-module by [4]. So A and 
/A/L" are also finitely generated over Z. Hence we have 
(3.2) charAA - charA (/A /£x) = charA.t;x 
(cf. e.g. [18, §2]). The Iwasawa main conjecture proved in [16] asserts the following: 
Fact 1. The torsion A-module x bas the characteristic polynomial P×(T): 
charA ' = Px (T). 
Hence the relation (2.2), charAA [ P(T), holds. Furthermore, Greenberg's 
conjecture for (p,)) is equivalent to the following: 
charA(/A/£ <) = Px(T). 
Next we recall results on the structures of the A-modules/A x and 1At/C  in [11], 
[6] and [18] (Fact 2), which are essentially used in the proof of our main theorems. 
(C < is a group of cyclotomic units defined below.) Since we are assuming f # 1, 
ch = NQ(çspn+)/(1 - Çln+) - is a unit in E, and hence c,  £n. We regard 
c as an element of £ ® O. We define x  O[A] by 
 :=  .()-1 . 
* ®X() -1 is an element of L' x. We can see that Nmm(Cm) = c 
Then c  = -«e c 
for all m >_ n >_ 0. Then we put 
c := (c)n>0 e/A x = lfi__m/A x 
and denote by C x the submodule of b/< generated by c over A. 
Let OE denote the Zv-torsion of/A and put OE := 1)__moE, where the projective 
limit is taken with respect to the relative norms. Then we have 
Tx -- (O/(p TM, 1 - Xo-l(p))) ®z» 7Lp(1), 

and hence 

(3.3) Ex  J'{l} if XW-I(p) # 1, 
[ A/(T - q) if X:a;-l(p) = 1, 

where X; (1) = /v"" 
The following fact plays an important role to prove Theorems 2.2 and 2.6. When 
the order of X is not divisible by p, this is the theorem of Iwasawa and Gillard ([11] 
and [6]). 
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Fact 2. ([18, Theorem 2.1, Proposition 4.2]) There is a natural A-homomorphism 
 "H x --A 
for which the kernel is "ffx and the image is (T - q, 1 - Xw-l(p)), the idem of A 
generated by T - q and 1 - \w -1 (p). rthermore, we bave 
 («) : (). 
Put ç :: > N,(H), with the norm Inaps Nm,n from km [o kn. In the 
following lemma, we determiue the structure of the A-modules ç to deal with the 
A-module H/. 
Lemma 3.1. 
(i) The projectzon   ç induces the following isomorphisms: 
[Hx/(Hx)  if x(p ) # 1, 
ç  [U/( u) q k(P) = 1. 
(ii) We bave an isomorphismU/F  O/(1 - 2(p)). 
By this lemma, the holnolnorphisln  in Fact 2 induces a A-homomorphism 
 follows: 
 : V  n/(0) 
where 0 = w (resp. oe) if X(P)  1 (resp. k(P) = 1). We further sec that the 
kernel of  is , and we can deternfine the cokernel by Fact 2. 
Pwof. We prove this lemma in a way similar to [6, Proposition 2] by using local class 
field theory. Let D be the decomposition group of p in A and put ï  .k[D. Fix a 
prime idem p of k over p. Write U for U,, the group of the principal units of k,, 
for ail n  0. We regard U := lin U and U as modules over Z,[D]r] = 
in a natural way (with  = 1 + T). Put V :=  Nm,(U) with norm map 
Nm, flore km, to k.. Then we have A-isomorphisms 
(a.a) u:  u2  o, v  v  o 
(oee [18, 4]). Here Op denotes the ring generated by the values of  over Zp. 
If X(P) = 1, i.e., ç = 1, then k = Q. In this case, the sertions (i) and (ii) 
follow from [6, Proposition 2], [20, Lemma 13.53] and (3.4). 
We sume that X(P)  1, i.e.,   1. Let  be the adic completion of 
 := lira " 
kn,p/(kn,ç) p, and let  := limon. Since the (local) cyclotolnic 
m 
extension k,p/kp is totally ramified, we have the following commutative diagram 

with exact rows of Zp[Dl[Fl-modules: 

(3.5) 

A 
1 --+ U --+ Xo --+ Zp  1 
1 --* Un -- Xn -- Zp  1. 

Here D and F act on Zp trivially. Let k ab be the maximal abelian p-extension of 
n,p 
kn, for 0 < n < oc. We have isomorphisms Ân  Gal(kb,/k,) and N)ç := 
fm> Nm,(m) Gal(kb,/k,)of 7,p[D]Fl-modules, by local class field theory, 
and an isomorphism 
Gai(k , k,) = Gal( /k,,)/WnGal( /k,, 
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holds. Then we bave N.,  )m/.ç". Therefore the kernel of ¢ in (3.5) is 
X. So the kernel of the projection U N 1/' is (X-) Clearly we have 
(X)  D (2£) . Let x  be an element of (.çoe)i, i.e., x e .ç Nz OD and 
(x) (a)-a = 1 for all a  D. Since .ç has no nontrivial element that is killed 
by w ([12, Theorem 25]), we have x {(a)-a = 1, that is. x  e (2) . Thus 
(.ç) = (). We also sec that 2ï = U i by (3.5) under the sumption 
 ¢ 1. Hence, the sertion (i) has been proved. 
To prove (ii), we first determine the structure of (NU)/ç , where NUn 
denotes  N,(U). If the order of ï is prime to p, the functor (  ) is 
exact,. Therefore, in this case, we bave V = (NUn) , that is, (NUn)/I  
OD/(1 -- (p)). Then we consider the case where p dMdes the order of ï. Decom- 
pose D = Dp x D' with the p-Sylow subgroup D v of D and put  = ï[D'.  use 
the following lemma. 
Lemma 3.2 (cf. [17, Lemma II.2]). Assume that Dp is ontviai and that [D 
is a faithful character of Dp. Let C be he subgroup of Dp of order p od  the 
norm 4 C in Zp[D]. Then, for a Zp[D]-module Al, we bave a Zp[D]-isomohism 
M   ker(Nc  el  eAl), 
 whe e &notes the zdempotent of Zp[D'] corspoding o . 
We sec that °(C, U) = 0 since kp/Qp is tamely ramified. Therefore, bv the 
above lemnm and the fact that the kernel of U  NU is X  proved above, we 
have an immorphisln 
(G)/V  °(c, (ç*). 
The fact that  h no nontrivial element killed v  implies an isomorphism 
°(C, (%)-)  °(C,.ç%). By using i(C.U) = 0 for i = -1,0 and (3.), we 
have isomorphisms °(C,«ç«)  °(C,eZv)  O/(1 - k(P))- Hence 
(u.)/v  o»/( - (p)). 
under the sumption ï  1. Thus, we have completed the proof.  
Finally, we shall show the ffeeness of x (Lemma 3.5). The following is well 
known. 

Fact 3 ([12, Theorem 181). 3¢ has no nontrivial finite Zp[A][rl-sub,nodule 
We need the following lemma, which follows from the Leopoldt conjecture for k 
and p proved in [1]. 
Lemma 3.3 (cf. [20, §5-51). 
(i) The inclusion E -- Sn induces an isomorphism 
for any a  O. 
(il)  is Zp-tOrsion free. 
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By the above lelnma, we can regard "11" as a submodule of fils and also of . We 
show the following lemma by using Fact 3. 
Lemma 3.4..1/'/1" bas no nontrivial finite Zp[A]F-submodule. 
Pro@ We identify Zp[/Xlr  with Ep[AIT  by 3' = 1 + T. For a Zp[A]T-module 
11I, we put Mr := {m  M I Tre = 0}. If I is nontrivial of finite order, then MT 
is nontrivial. Thus, by Fact 3, it suffices to show that (XOE)T  T. Since çT is 
trivial, we bave an exact sequence 
1  T  (-/OE)T  (OE  TX)/T  1, 
where v is induced by x  Tx for x   Let M0 be the mimal abelian extensiou 
unramified outside p. The restriction map   Gal(Mo/k) induces a surjection 
T  T0, and hence an isomorphisln 
T/(Tn TX) 
On the other hand, we see that OE is trivial or p,+ @z Zp[/D], where D denotes 
the decomposition group of p in L for all   0 according to whether OE0 is trivial 
or hot. Thus we bave 
Therefore we have OE  T,E = ,  dcsired. 
Using Lemma 3.4, we prove the following: 
Lemma 3.5. (S ) is a principal ideal generated by char(/Sx). In particular, 
S  is a free A-module of tank one, 
Proof. There exists an element f(T) of A such that the priucipal ideal (f(T)) of 
A has a sublnodule (S ) of finite index. We first show that f(T) is contained in 
 () = (T - q, 1 - k-(p)). If k-(p)  p,p  {1}, this sertion is clear since 
 () is a principal idem of A. Assume k-(p)  p,p  {1}. By Lemma 3.1 (i) 
and Ft 2, we have 
 N=,()(U)-/(U)   ((),(U))/.(U). 
For suciently large n  0, (S )  w(f(T)). Hence the above A-module has a 
finite submodule isomorphic to (f(T), ())/(). However, S is Zp-torsion 
free (Lemma 3.3 (ii)). Thus we have f(T)  (). 
By Lemma 3.3 (il), S x  OE = { 1 }. Then /S x has a A-submodule isomorphic 
to OEx, which we also denote bv OEx. By Fact 2, we have an exact sequence 
Then (/Sx)/T x has a finite A-submodule isomorphic to (f(T))/(Sx), since we 
have proved that f(T)  (T-q, 1 - kw-(p)). By (3.1), we can regard 
as a submodule of X/OE. Therefore, by Lemma 3.4, we bave (f(T)) = 
Furthermore, by using the above exact sequence and (3.3), we have (f(T)) = 
(char(/gx)). We have proved the lemma. 
Remark 2. By using Fact 1 (the Iwasawa inain conjecture), Fact 2 and the above 
lemma, we tan sec that Greenberg's conjecture for (p, g) is equivalent to the fol- 
lowing: 
gx = C . 
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4. PROOF OF THE MAIN RESULTS 
We rewrite the condition (Hp, n) as follows: 
Lemma 4.1. For each n _ O, the condition (Hp, n) is equivalent to the following 
condition: 
" rnp, n 
(up,) c P,n ¢ () . 
Proo.f. We fix,  0. The isomorphism in Lemma 3.3 maps the cls [Y'] in 
n n to the cls [cn (-«)xe..] in nn . Thus the condition (Hp n) holds 
if and only if 
Putting  = ker(E a  $  $), we have e$F ' = e, and 
 . Then the above condition holds if and only if 
e(1-)Xp  
By using Lemma 3.2, we ao see that ff is isomorphic to  by x   (x 1) for 
x   ff. (Note that  is in O[A].) We can see that xp, = xXp, n. Therefore 
the above condition is equivalent to the condition (Hp,). 
Proof of Theorem 2.6. ç shall show that P(T) [ charhA holds if and only if the 
opposite 
of (P,n) holds for all n  0. 
We put Q(T) = Px(T)/P(T). By Fact 1 (the Iwawa main conjecture) and 
(3.2), we have 
charA (N /$x). charA A = P (T). 
Then P(T)  charAA if and only if charA(N/$ x)  Q(T). By Lemma 3.5, the 
latter condition is equivalent to saying that Q(T)  ç($). Let Q(n) denote 
Q(T) mod in t/(). Here we recall that  is wh (resp. Yn) if (p)  1 
(resp. X(P) = 1). By using the A-homomorphism n " ç  A/() defined a.er 
Lemma 3.1, we have Q(T)  () if and only if 
for all n  0. For a fixed n  0, it suces to show that (4.1) is equivalent to 
(p,). Since A/() is Z-torsion free, the condition (4.1) holds if and only 
(4.) ,Q()  ((  
We defined Xoe, n to satisfy mp, nQ(T)  Xp, n(T)P(T) mod . By Foet 2, we 
have çn(c ) = g(T) = u(T)P(T) where u is the unit of t appearing in (2.1). 
Hence we have 
(4.) ,Q() = (c»,). 
By Foet 2 and Lemma 3.1 (i), the kernel of n is OEn. Hence, the condition (4.2) 
holds if and only if 
(4.4) c xP'  (  
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If we assume that the condition (4.4) holds, there exists e E $ ç V} such that 
(cx'n/e'") e "F. Since c  e £ and £x ç V, = {1} (Lemma 3.3 (ii)), 
c xe'n = e'e,n. Thus the condition (4.4) holds if and only if 

(4.5) 

In the case where Xw-l(p)  pp, we have (//x) = A by Fact 2. Then, clearly 
Q(T) E (//x), and hence Q(') E (F) for all n _> 0. In this case, we have T = 
{1}. Thus, by (4.3), we obtain c xe' e (F) 'e.. Therefore, the condition (4.5) 
is equivalent to (7-/p,,). If we assume Xw-l(p) 6 pp, then X(P)  Pp. Hence, 
by Lemma 3.1 (il), we have V =//. Thus, ($)'e. ç (F) 'e, = ($)'e.. This 
completes the proof. [] 

Proof of Theorem 2.2. The assertion follows from Theorem 2.6 and the following. 

Lemma 4.2. Let P(T) be a factor of P×(T). Then we have that P(T) satisfies 
(T) if and only if c x x,., ¢ (bl)me,'Tn for some n >_ O. 

Proof. By the definition, P(T) satisfies (T) if and only if Q(T)  (T-q, 1-Xw - (p)) 
with Q = P×/P. By Fact 2, we have (//x) = (T-q, 1 - X(P)). Hence P(T) 
satisfies (T) if and only if Q(T) ¢ (//). Furthermore, by using an argument in 
the proof of Theorem 2.6, we can show that Q(T) ¢ (//<) holds if and only if 
ch  (v)me'n for some n _> 0. By Lemmas 2.3 (i) and 3.1 (ii), if P(T) 
satisfies (T), then V =//. Thus, the lemma has been proved. [] 

Remark 3. Assume P(T) satisfies (T). Then we can directly show that P(T)  
char^A. Indeed, we have Q(T)  q(L/) with Q = Px/P by the assumption and 
Fact 2, and hence Q(T) ¢ q($×). The latter condition is equivalent to saying that 
P(T)  char^A by Fact 1, Lemma 3.5 and (3.2) (see at the beginning of the proof 
of Theorem 2.6). 

5. BASIC FACTS ON THE X-PARTS OF THE IDEAL CLASS GROUPS 
In this section, we shall show suflïcient conditions for A to be trivial (Lemma 5.1 
and Remark 1), which are more or less known. We use the same notation as in the 
previous sections. 
For a Zp[A]-module M, we define an (9-module Mx, the x-quotient, by 
M := (M ® O)/AM ®, O), 
where I x denotes the idem of (9lA generated by ail elements of the form  - X(), 
 G A If M is finite, then M x and M x have the saine orders (cf., e.g., [18, §2]). 
Thus we have 

(5.1) #A = #A«. 

We first show the following: 

Lemma 5.1. Let X be a (not necessarily even) Dirichlet character with X(P)  Pp. 
If A is trivial, then, for all n >_ O, A are trivial, and so is A. 
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Pro@ By Naka.vama's lelnnm and (5.1), it suffices to show that 
A,,x -- A,x/wA, x 
for each n >_ O. Let L be the lnaxilnal unramified abelian p-extension of k, L 
the nmxilnal abelian extension of kn contaiued in L, and Hn the Hilbert p-class 
field of k for each n _> O. By class field theory, we have A, - Gal(H,/kn) 
and A  Gal(L/k). We further see that Gal(L/k)  A/w,A, and hence 
Gal( L,/k ) x " A ,,,x /w,A.,,x. Then we show that Gal( L,/k )x " Gal( H,/k,)x. 
For a prinle ideal p of kn over p, let Ip denote the inertia group of p in Gal(Ln/kn). 
Since L,/I,', is unran,ified outside p. we have al, exact sequence of Zp[Gal(kn/Q)]- 
nlodules 
H Ip  Gal(L/k,)  Gal(H,/k)  O. 
P[P 
We further see that 1-Iplp Ip - Zp[A/D] (recall that D is the decomposition group 
of p in A). We recall that A' is the priule-to-p part of ,_X, b = \l,x' and e 
is its idempotent. By the assulnption that \(p) ¢ pp, i.e., b(p) ¢ 1, we have 
eW(Zp[A/D]) = {1}, and hcnce e¢Gal(L/k) - e«Gal(H,/]%). Therefore we 
obtain Gal(L,/k,), " Gal(H,/k,), (see [18, §2]). Since Gal(k/k)× = {1}, we 
have proved the claire. [] 
Applying Theorelns 2.2 and 2.6 to verify the conjecture for \, we ha,ce only to 
consider the case where \(p)  pp or A is nontrivial by the above lemnm. We 
give a sufficient condition for A to be nontrivial. 
Lemma 5.2. Let \ be a Dirichlet character and d the prime-to-p order part of 
\. If eAk, is nontrivial, then so is A, where Ak, is the p-Sylow subgroup of the 
ideal class group of kb. 
Pro@ Assume that eAk, is nontrivial. Let p be the Dirichlet character with X = 
bp. Since kp/Q is a cyclic extension of degree a power of p, at least one prime 1 is 
totally ralnified, and hence a prime ideal of kw above I is also totally ramified in k, = 
k¢kp. Then we see that the norm map from the p-Sylow subgroup of the ideal class 
group A,, of k x to A, is surjective by class field theory. Therefore, (eAG)x 
eAk, is also surjective; so (eA M)x is nontrivial. Let @ denote a generator of 
is the p-Sylow subgroup Ap of A and put p = X]%- By the isomorphism 
A0,x = (A)x  (e,l,A× ®o' O)/(Sp - p(Sp))(e,l,A × ®o' O) 
(cf. [18, §2]), we have 
((eAM)a , ®o' (.9)/(1 - p(6p))((evAG)a, ®o' (.9) - A0.x/(1 - p(6p))Ao.x, 
where (.9  denotes the ring generated by the values of , over Zp. Therefore, Ao.x is 
nontrivial, as desired. [] 

Finally we shall show that if VO(tl,xw-x) = 0, then A are trivial for all n >_ 0 
(Remark 1). Let k×-a denote the fixed field of the kernel of the odd Dirich- 
let character wX-1; so wX - is a character of Gal(k×-,/Q). The p-Sylow sub- 
group of the ideal class group of the n-layer kx-l,, of the cyclotonfic Zp-extension 
kwx-l,oo/kwx- is a Zp[Gal(k,x-/Q)]-module and we define A x- by its w\ -1- 
part for each n >_ 0. As a consequence of the Iwasawa nmin conjecture proved in 
[16], the following was proved. 
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Theorem 5.3 ([16, Theorem 2 in §1.10], [17, Theorem II.1]). Let X be a nontrivial 
even Dirichlet character. Then we bave 
#Aô - = #(O/BI«-,O). 
We know the following reflectioi1 theorem (cf. [20, Theorem 10.9]). 
Lemma 5.4. Lai  be an aven Dch[e{ charac{er. Then the order of A,/A,x 
divides that of A- /A - for each n  O, where  is a prime dament of O. 
x- and A 
Assume vo(B,x-) = 0. By the al)ove and (5.1), we have that --0 
e trivial. For any character , we have kw-l(p)  p or k(p)  p». Then, 
by Lemma 5.1, A x- or A are trivial for all n  0. In the formcr case, the claim 
follows from Lemma 5.4. 

6. EXAMPLES 
In this section, we verify Greenberg's conjecture for several exanlples by using 
Ç, orollary 2.4 and Theorem 2.6. The COlnputations in this section was carried out 
by Kazuo Matsuno whom the author wants to thank. 
Corollary 2.4 asserts that the condition that Px(T) is irreducible or that 
vo(Bl,x,-i ) = 0 implies A = 0 when \w-l(p) E pp \ {1}. V'e give sonle 
exmnples satisfying the condition. 
The above condition is clearly satisfied when A\ = degP\(T) = 1. hl [19, 
Lemmas 3 aud 4], the author showe(t t.hat, for each even Dirichlet character b of 
the first kind, there exist infinitelv lnany characters p of p-power order such that 
p =  and p(p)  1. Then ( = bp satisfies the conditiou in Corollary 2.4 if  -- 
1 and bw-l(p) = 1. For exalnple, these conditions are satisfied if p = 3 and b is 
the quadratic character of conductor 33 (cf., e.g., [2] for other exanlples). Therefore 
there exist infinitely many characters ) satisfying the condition in Çorollary 2.4, 
and hence A x = 0. In [18], a method for finding infinitely mauy x('s satisfying 
vo(Bl,x,-, ) = 0 and xw-l(p) E pp \ {1} was also given. However, for such 
characters X, we can also verifv that A x = 0 by Leulma 5.1 instead of Corollary 2.4 
since we see that A = 0 if A x =- 1 or vo(Bl.x,-) = 0 (for the latter see §5). So 
we next give examples such that A is nontrivial aud Px(T) is irreducible. 
Let p = 3 and b be the quadratic character of couductor 321 = 3  107. For 
a character p of order 3, we put ,( = bp. V'e see that e¢Ak, = AQ(3x/T) is 
nontrivial (the class uumber of Q(3x/-3--) is 3), and heuce A is also nontrivial by 
Lemma 5.2. Let p be a character of order 3 whose conductor is 7. Then p(3)  1; 
so xw-l(3) ¢ P3 \ {1}. For this \, we calculate P)t(T) nlodulo a power of p as 
in [10] and [5] by using the fact that the Stickelberger elements produce gx(T) 
and check the irreducibility of Pk(T). More precisely, we calculate NPx(T ) = 
Px(T)Px-,(T )  Z3[T] 0( -1 is a Q3-conjugate of /), since it is easier to handle 
Z3-coeflïcient polynomials than O = Z3[Ç3]-coeflïcient ones. \Ve have 
NPx(T ) =- T 4 + 5130T 3 + 1020T 2 + 2214T + 4977 nlod 3 s. 
We see that NPx(T) is irreducible over 7, 3. So Px(T) is also irreducible over O. 
Thus, this X satisfies the condition in Corollary 2.4, and we have ),, = 0. Moreover, 
this implies that Greenberg's conjecture for kx, the cyclic field of degree 6 whose 
conductor is 2247 -- 3- 7- 107, is true by Lemma 2.1. In fact, Ichinlura and Sumida 
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[10, Proposition] showed that A = 0, and we can see that Ap = 0 by Iwasawa's 
emma («. [20, Lemma 10.4]) since p(3)  1. 
We fllrther check the irreducibility of P,p(T) for a nontrivial character p of cyclic 
cubic fields kp with conductor less than 10 3. There are 128 such fields, and 86 fields 
among them satisfy p(3) -# 1 (i.e., 3 does hot decompose in those fields). Among 
these 86 fields, P,p(T) is irreducible for those p whose conductor is in the following 
list: 

7, 43, 97, 109, 127, 139, 157, 181, 211, 229, 277, 337, 
349, 379, 409, 607, 691, 709, 733, 739, 751, 877, 907, 
217 = 7- 31, 301 = 7- 43, 427 t = 7.61, 469 t = 7- 67, 
511 t = 7- 73, 553 = 7.79, 721  - 7. 103. 

We remark that there are two cyclic cubic fields kp satisfying p(3) -# 1 whose 
conductor is one of the lmmbers with t in the above list. (We have only one such 
field for others.) Among them, P,p(T) are irreducible for both fields of conductor 
427,511,721 and for one of the fields of conductor 469. Thus we obtain 33 cyclic 
cubic fields kp of conductor less than 103 such that ,( = Cp satisfies the assumption 
of Corollary 2.4. 
We can see that Ap = 0 if the conductor of p is prime and p(3) -# 1 by Iwasawa's 
lemma. Furthermore, Yamamoto kindly informed us that we know that Ap = 0 for 
p whose conductor is 217, 301,427, 469 or 721 (see below for 553). Thus, for such 
p's, Greenberg's conjecture for k x = kp(3x/-3-) is true by Lemma 2.1. 
We apply Theorem 2.6 to several examples. 
Let p = 3, k be a unique cubic field of conductor 553 = 7- 79 in which 3 does hot 
decompose and X a nontrivial character of k. We fix a generator a of Ap (= A) 
and put :(a) = Ç, a nontrivial p-th foot of unity. We have 
P(T)  T- (122 + 160ç) mod 3 . 
We check the condition (Hp,,,2). We see that mB,,,2 = 33 = 27 and 
P,2 -- T s + (23 + 25Ç)T 7 + (3 + 3Ç)T 6 + (24 + 15Ç)T 5 
+(6+3Ç)T 4+9ÇT 3+(21+9Ç)T 2+(24+21Ç)T+9+18Ç mod33 . 
Let l = 29863. Then a prime ideal ! of k2 above I is of degree one. We verify that 
c -«):e' mod | ¢ (Z//Z) ×27, and so we have A = 0. Greenberg's conjecture for 
k x (and kx( ) also) is also true. 
Let p --- 5 and ¥ be a character of order 5 with prime conductor f. By a re- 
sult of Kurihara [15, §4.4], we know that A< = 0 for all f < 10  except f = 
38851,41201,84551. We consider the case f = 38851. By computing NP,(T) = 
4 
1-L=I Px'(T) Ç Z5[T], we see that Px(T) decomposes into a product of two irre- 
ducible factors P(T) and P2(T) of degrees 1 and 3 respectively. In [15], Kurihara 
introduced an invariant  associated to X and showed that, for a factor P(T) of 
Px(T), we have P(T) ¢ charAAe if deg P(T) >_  (cf. [15, Remark 1.11]). We have 
 --- 2 in this case. So we check the conditions in Theorem 2.6 only for the factor 
P(T) of degree 1. We have 
Pi(T) -- T - (1313 + 416Ç + 1834Ç 2 + 2427Ç 3) mod 55. 
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We check the condition (Hpl,1). We see that mp,l : 5 and 
T 3 
'p , = + (3 +  + 402 + 23)T 2. 
We verify that cï -°)p''I mod ! ¢ (Z/lE) ×5 with I = 9712751, where a prime ideal 
[ of kl above I is of degree one. 
We further consider the cases where f -- 41201, 84551 in the above setting. 
We see that Px(T) decomposes into a product of two irreducible factors P1 (T) 
and P2(T) of degrees 1 and 3 respectively, in both cases, and have n = 2 (resp. 
3) if f = 41201 (resp. 84551). Hence we have only to check the conditions in 
Theorem 2.6 for the factor Pi(T) of degree 1 for both frs. We verify that the 
condition (Hp,,l) holds in each case. Therefore, together with the result of Kurihara 
[15, §4.4], we have that A x = 0 for p = 5 and ail primes f < 105. 
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PSEUDO-HOLOMORPHIC CURVES 
IN COMPLEX GRASSMANN MANIFOLDS 

XIAOXIANG JIAO AND JIAGUI PENG 

ABSTRACT. It is proved that the K/ihler angle of the pseudo-holomorphic 
sphere of constant curvature in complex Grassmannians is constant. At the 
saine time we also prove several pinching theorems for the curvature and the 
K/ihler angle of the pseudo-holomorphic spheres in complex Grassmannians 
with non-degenerate associated harmonic sequence. 

1. INTRODUCTION 

In this pal)er we study confornlal lninimal two-spheres in complex C, rassmann 
nmnifolds by using the harmonie sequence. Given a harnlonic map ç of surfaces BI 
into the complex Gra.ssmannian Gt-,,, by using the 0'-transfornl Chern and Wolfson 
([3], [10]) obtained the following harmonie sequence associated to ç: 
O' O' O' O' 
ç=ç0----) çl  "'" ---- çj  "'" , 
where çj+l = Otçj, j = 0, 1,---, and ç-  111 -- Gk,, are harmonic maps, kj = 
rank(ç-). If ça is anti-holomorphic, then kj+l = 0. When ç is holomorphic we 
call çj a pseudo-holomorphic curve generated by ç. Such curves with the induced 
metrics from the associated complex Grassmann manifolds form a class of minimal 
imnlersions. When hj = kj+l we say that çj is non-de9enerate. When k- = kj+ for 
all j we say that the harmonic sequence associated to the nmp ç is non-degenerate. 
When specialized to G I., = CP '-, any pseudo-holonlorphic curve is obtained 
from a holomorphic curve projected into CP '- Calabi ([2]) showed that any 
simply connected holomorphic curve in CP '-1 is completely determined, up to 
holonlorphic isometries of CP '-1, by its induced metric. Calabi also showed 
that a simply connected holomorphic curve of constant curvature in CP '-1 is 
the Veronese curve, up to unitarv equivalence. For a pseudo-holomorphic curve 
in CP '-, Bolton, .Jensen, Rigoli and Woodward ([1]) showed that, up to a holo- 
nlorphic isonletry of CP '-1 file harmonic sequence determined by an3" linearlv 
full conformal nfinimal immersion of constant curvature in CP '- is the Veronese 
sequence, in which each map is a minimal inlmersion with constant curvature and 
constant Kiihler angle. 
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It is well known that the rigidity fails for pseudo-holomorphic curves or holo- 
morphic curves generalized to Gk., ([5], [14]). For exanlple, Chi and Zheng ([5]) 
classified the holornorphic curves of the Riernann sphere into G2,4 with the induced 
constant curvature 2 into two classes, up to unitary equivalence, in which none of 
the curves are congruent. Let ç : S 2 -- Gk,, be a pseudo-holomorphic curve in 
a complex Grassmannian G,n. Problem: Is the tçïihler angle O(ç) of ç constant 
when its Gauss curvature K(ç) is constant? What are the relationships between 
the hïihler angle and the Gauss curvature ofç and its ramification index? In this 
paper we will investigate these questions. 
In the second and third sections of this paper we obtain sorne fundamental 
fornlulas for pseudo-holonlorphic curves in complex Grassmann manifolds. 
In the fourth section, by nsing these fornlulas we prove that the curvatures of 
pseudo-holomorphic curves are equal to 4IN (N is a positive integer) if these cur- 
vatures arc constant (this result was proved by Chi and Zheng in [5])(Theorem 4.1), 
and prove that Kfihler angles of pseudo-holomorphic curves of constant curvature 
are constant (Theorcnl 4.2). In this section, we also give a harmonic sequence, 
in which each map is a nlininml immersion with constant curvature and constant 
Kihler angle. 
In the final section, we give some pinching theorems for pseudo-holomorphic 
curves with the associated non-degcncrate harmonic sequence for curvatures and 
K/ihler angles (Theorenls 5.2, 5.6 and 5.7). At the saine tinle we also show that 
the K/ihler angle of a pseudo-holonlorphic curve is iudependent of its ramification 
index under the assumption of Theorenl 5.2. 

2. MINIMAL IMMERSIONS AND HARMONIC SEQUENCES 
Let U(n) be the unitary group. Let M be a sinlply connected domain in the 
unit sphere S 2 and let (z, ) be a complex coordinate Oll AI. ,Ve take the metric 
ds = dzd5 on AI. Denote 
0 0 1 1 
0 = --z' = --z' A.. = -s-Os, A= -s-s. 
Let s  M ---, U(n) be a smooth map; then s is a harmonic map if and only if it 
satisfies the following equation ([9]): 
(1) OA = [A:,A]. 
If s  S 2 -- U(n) is a harlnonic map, then s is a conformal map; so s is a minimal 
immersion. Let w = g-ldg be a Maurer-Cartan form on U(n), and let dsU() = 
! tr ww* be the metric on U(n). Then the metric induced by s on S 2 is given by 
8 
(2) ds  = -trA,AdzdS. 
Let G,, be the complex Grassmann manifold consisting of ail complex k-dimen- 
sional subspaces in C'. Here we consider G,, as the set of Hermitian orthogonal 
projections onto a k-dimensional subspace in C', i.e., G,, = {ç is the Hermitian 
orthogonal projection onto a k-dimensional subspace in C'}. Then 
is a Hermitian orthogonal projection onto a k-dimensional subbundle z/C S 2 × C', 
and s -- ç - ça_ is a map from S 2 into U(n). It is well known that ç is harmonic 
if and only if s is harmonic. If ç±ç = 0 or ça_c3ç = O, we call ç a holomorphic 
curve or an anti-holomorphic curve in Gk,,. 
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Using ç, the harmonic sequences (see [3], [10]) are given by 
O' O' O' O' 
(3) ç = çO -- çl -- "'"  ça -- "" " , 

0" 0" 0" 0" 
(a) -- ç0  ç-1 --- --.  ç-ct --- , 
where ça " S 2 x C n  Im(ç_10ça_l) and ç-a " S 2 
are Hermitian orthogonal projections,  = 1, 2,.-- . 
Proposition 2.1 ([7]). For (3) and (4), we bave 
If ç0 is a holomorphic curve in (3) or an anti-holomorphic curve in (4), then 
elements in (3) or (4) are finite and are mutually orthogonal. If there exists a 
holomorphic curve ç0 in G,n such that ç is an element in the harmonic sequence 
(3), i.e., ç = ça " S 2  G,a belongs to the harmonic sequence 
O' O' O' O' 
(5) 0  çO  çl  "'"  ç = ç«  "'"  ç«o  0, 
then we call ç a pseudholomorphic rve in complex Grsnann manifolds, and 
0 is called the length of the harmonic sequence (5). 
Now we sume that ç = ç  S 2  Ga,,n is a pseudeholomorphic curve. 
Then we may choose the local unitary frame el,e2,.-. ,en on S 2 x C n such that 
ek«_+l,''" ,ek span Inl(ç_10ça_l), where ka = rallk(ç_10ça-l),  = 
1,2,---, k0 = rank(ç0). 
Let Wa = (ek_+l, ek«,''" ,ek) be an (n x ka)-matrix. Then we have 
K * 
(6) ç, = - 
(7) W2W = I., ,+1 = 0, W2W_I = O. 
By (7) and a straightforward computation we obtain 
{OW = 1%+1 + W, 
(8) W = -_i;_i - 
where a is a (ka+l x ka)-matrix and a is a (ka x ka)-matrix,  = 0, i, 2,.--. 
It is well known that a = 0 or a-1 = 0 in (8) if and only if ça is anti- 
holomorphic or holomorphic. It is very evident that integrability conditions for (8) 
are 

(9) 
(10) 
By (8) A ") and A (a) for ça are given by 
(11) A 
(12) 
It can eily be checked that (9) is equivalent to (1). An immediate consequence of 
(S) 
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Proposition 2.2. Let  = ça  S 2 --, Gk,,,be a pseudo-holomorphic curve, with 
a and q2a determined by equations (8). Then a and q2a satisfy equations (9) 
and (10). 
Let ç(a) = ço @ "'" @ ça for (5) and k(a) = ko + --- + ka. Then by Proposition 
2.1 we have 
(13) 
= 
[Ience ç(a) . S 2 _ Gkt),  is a holonorphic lnap, and the harmonie map sequence 
(5) becomes 
0' ç(a) 0' 0' 0' 0' 
(14) 0  - ça+l - "'" - çao - 0. 
If/ca 
is non-degenerate for a = 0, 1,--- , c0 - 1 in (5), i.e., k0 = bi = .... kao, then the 
harmonie scquence (5) is callcd the non-de9ezterate harmonie sequence associated 
to thc harnlonic nmp ç = ça- Now we assunle that ça is non-degenerate; then 
S 2 
dct(la) is a wcll-defined invariant on raid has only isolated zeros. Let 
(15) la = tr(lt). 
Then 

and we bave 
Proposition 2.3. If ç = ça : S 2 -- Gko,, is a non-degenerate pseudo-holomorphzc 
curve, then 
(16) 20log [ det(a)[ = la-1 - 21 + la+l. 
Proof. By (9) and the rule of differentiating a deternlinant, we get 
 log dct(2a) = tr(202) = tr +1 - tr ,, 
01ogdct()  -1. . 
= tr( 
It is hOt diflïcult to obtain (16) by (10). [] 
Remark. If ça is non-degenerate for ail c in (5), then 
(17) 20Olog I dct(a)[ 
for c = 0.1,---,0- 1, where l-1 = la0 = 0. When ha = 1 for ail a, then 
la = I det(a)l 2, and (17) is just the unintegrated Pliicker formulae for la derived 
by Bolton, Jensen, Rigoli and Woodward in [1]. 

3. K)i.HLER ANGLES AND GAUSS CUPVATUnES 
If ç : /I --, Gk, is a conformal immersion of a Riemann surface/I, we define 
the K£hler angle of q to be the function 0 :/I - [0, 7ri given in terres of a complex 
coordinate z on M by 
O(p) Idç(O/O5)l 
(18) tan 2 ]dç(O/Oz)]' P  1I. 
It is clear that 0 is globally defined and is snlooth at p unless O(p) = 0 or u. Let 
z - x + x/-Z]-y, and let J denote the conlplex structure on Ga,; then 0 is the 
angle between Jdç(O/Ox) and d(O/Oy). The importance of the K£hler angle in 
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the theory of minimal immersions of surfaces into K/ihler manifolds was pointed 
out by Chern and Wolfson [4]. Indeed,  is holomorphic if and only if O(p) = 0 for 
all p E M, while ç is anti-holomorphic if and only if O(p) = ?r for all p E M. 
Now suppose that ç  S 2 --, Gk,, is a conformal minimal immersion in the 
harmonic sequence (5). Then each ç«  S 2 --, tSlk,, is a conformal minimal 
immersion. So there exists a finite set X« (see [1]) such that the K/ihler angle 
0o- s 
is well defined, and is smooth on S 2 
Let t = (tan) . Then, in terins of a local complex coordinate z, 
Idç(O/O)I 
(19) 
Let ds and ds«) be the metrics on S 2  X induced by ç« and ç(«) respectively. 
Then by (11), (12) and (13) we have 
(0)  = (t._, + .)ed, .) = .. 
The Laplacians « and («) for ds and ds«) are given by 
(21) 
and the curvatures K«, K(«) of ç« and ç(«) by 
2 2 - 
(22) K« = I«-1 + 
the area forms dv« and dv(«) by 
(23) dv« = (I«_ + l)  
dz 
dz 
Choose holomorphic sections fl,"" , 
and 
(23) f A--. A £()  S 2  C ((«)) 
is a nowhere zero holomorphic curve. 
Let F («) = £ A--- A £(,). Now consider the Plficker embedding (see [12], [13]) 
n --1 
(2«) [(")]-s   cP((«, ) , 
which is a holomorphic isometry, and 
(:5) 
Cpkk(E)] - 
By [1], we have 
(26) Ng 
and the degree 5« of F («) is given by 
(27) & - 2  l°gl()I A dz - 2  lA dz, 
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which is equal to the degree of the polynomial function F («) in z. We call 5« the 
degree of the holomorphic curve qo («). Thus from (17) and (27) we get 
Proposition 3.1. If qo = qo« : S 2 -- Gk,,n is a non-degenerate pseudo-holomorphic 
cue, then 
(28) -« = G- - 2G + G+, 
where « --    O log [ det « 1 A dz is the number of singular points of 
, i.e., the number of zeros ofdet 
Remark. If « is non-degenerate for a = 0, 1,--- , ao - 1, then -« = 6«_ - 26« + 
6+ for 1 a, and 6_ = 6«o = 0; in particular, when ko = .... k« o = 1, (28) is 
the global Plficker for,nula (oee [61). 
Let ds 2 = ]det«]2dz = «a, where « is a type (1, 0) analytic 1-form. 
Then ds 2 = «  a is a singular Hermitian metric. Let Ds =  ordp(«)p be 
pS 2 
the singular divisor of (S 2, ds2), i.e., the zero divisor of a- By the GausBonnet- 
Chern theorem we bave 
where ça = deg Ds. 
Ve say that ça is the ramification index of ça. Evidently, ça is a non-negative 
i.tCg¢. I . = 0, ç. i ¢¢0 .iyia by oto. et . ([1]). 
Let (5) be the non-degenerate harmonic sequence; if wa = 0 for a = 0, I,--- , a0- 
I, the harmonic sequence (5) is called totally unramified. Let ç = ça : S 2  
G, be the pseudholomorphic conformal immersion with the non-degenerate 
sociated harmonic sequence (5); we say that ç is a totally unramified pseud 
hdorphic cformal immersi if ç0,"" , çao is totally unrified. 
If ça : S 2  G.,. is a conformal minimal immersion with constant Khler 
angle, then we bave 

(29) t« -- 
and from (19) and (22) it follows that 
2 
(30) '« = 

O log 1«_ 1 = 2 O log 
lc,-1 + lc, le,-1 + 

4. CONFORMAL MINIMAL IMMERSIONS WITH CONSTANT CURVATURES 
It is well known that any complex submanifold of a (simply-connected, com- 
plete) space of constant holomorphic curvature is completely determined, up to 
holomorphic isometries of the ambient space, by its induced metric (sec [2], [8]). 
The Veronese sequence is the harmonie sequence 
O' O' O' 
(31) 0 ----* ço ---)  "" ---* ç, ---) O, 
where n = deg(o), and each ç« = [g,o,"", g«,,] : S 2 --' CP ' is given by 
g"'S--(l+z)« zS-«E(-1)k a-kJ n-Jk (z)' a,j=O, 1,...,n. 
k 
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Each map « in the Veronese sequence (31) is a conformal minimal immersion with 
constant curvature 
4 
(32) h'(«) = 
n + 2a(n - 
and constant K/ihler angle 0« given by 
( _)2 (n-+ 1) 
(33) tan = ( + 1)(n- )" 
Bolton, Jensen, Rigoli and Woodward ([1]) showed that, up to a holomorphic isom- 
etry of CP n, the harmonic sequence determined by  : S 2  CP ", which is a 
linearly full conformal minimal immersion of constant curvature, is the Veronese 
sequence. It is very complicated for pseudo-holomorphic curves in complex Grass- 
mann manifolds; for example, rigidity fails, but we still believe that there are some 
good geometric properties. In this section we discuss pseudo-holomorphic curves of 
constant curvature in complex Grassmann manifolds, and K/ihler angles. 
Let « : S 2  Gk,n be a pseudo-holomorphic curve with constant curvature. 
Then we know that 
[ff(a-l)]. S 2 __ 
are two holomorphic curves with degrees 5a- and Sa respectively. Consider the 
tensor product of [F («-1)] and [F(«)], 
T (a)_-_F( a-1)®F (a). 

(34) 
Then 

[T(«)]  S 2  Cp(k(.-l,)(¢,.-l,) -1 
is a well-defined holomorphic curve, and from (25) the metric induced by [T («)] is 
given by 
[T(«)]*ds 2  n   _ =[F(«-D]*ds 2   +[F(«)]*ds 2   
Cp(k(a«l)/[k(a-l)] Cp(k(a-l)) -I Cp(k(a))-I  
(35) [T(«)] *     -i = («- + 
Hence the curvature h% of « is equal to the curvature of [T(«)]. om [1], an 
immediate consequence is 
Theorem 4.1. If  : S 2  , is a pseudo-holomohic cue with cotant 
curvature K(), then K() = 4IN, where N is a positive integer. 
This theorem was proved by Chi and Zheng ([5]) by the method of the moving 
frame. In the following we will prove 
Theorem 4.2. If « : S 2  , is a pseudo-holomohic curve with constant 
curvature K«, then the Khler angle « of « is constant. 
Proof. om (22) we bave 
(3) K.(«_ + «) = -Oig(«_ + 
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When Ka is constant, from (22), (23), (27) and the Gauss-Bonnet theorem it follows 
that 

4 
(37) Ka -- 
Hence from (22) we obtain 

(38) 

2 4 
la 0 log(/a-1 + la) -- 

Choose a complex coordinate z on S 2 \ {pt} so that 
(39) la-, 4- la -- 
(1 + z$) 2" 
From (38), (ag) and (26) we obtain 
- ]F(a-1)]2JF(a)]2 --0. 
(40) 001og (1 + z) ao-'+ao -- 

Since we can choose holomorphic sections fl,"" , fk(o) in F(S 2 x C m) such that 
the maps F (a-l) and F (a) are polynomial flmctions on C of degrees 5a- and 
a respectively, it folows that (1 + z) 
non-zero constant limit c,  z  oe. So from (40) we get 
IF(-')]IF()I 
 C. 
(1 + z) 
Then we have 
]E("-l)l= = ¢_,(1 + z) a--', ]v(")l = = ¢(1 + z) «, 
where ca_l and c are constants. 
Hence, la-1 - (1 + z) 2 and la (1 + zg) 2' namely, ça is of constant curvature 
and constant Kghler angle. 
Dom (19) and (22) we know that if ç - S 2  Gk, is a pseudo-holomorphic 
1 
curve with constant Kghler angle 0, then K - 1 + ta K(). 
Remark. We do not need to sume that ç  S 2 + Gk, is non-degenerate in 
Theorem 4.2. 

To conclude this section, we give an example. This example is a harmonic 
sequence, in which the Gauss curvature and the Kghler angle of each element are 

constant. 
Let fo(z) = (1, 0, x/z, 0, z 2) and go(z) = (0, 

1, 0, z, 0); then 

/ 1 0 vfz 0 z 2 ) 
1 0 1 + z 0 z(1 + z) 0 
ço - (1 + z) 2 v 0 2z 0 vz2 
0 (l+z) 0 zg(l+z) 0 
72 0 VfZ 2 0 Z2 2 

determined by fo(z) and go(z) is a holomorphic map. 

 S 2 - G2, 
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An immediate computation shows that 
( 2 0, v(1-zS) 0, 2z ) 
fl(z,5)=ço(Ofo(z))= l + z' l + z5 ' l + z ' 

( 5 O, 1 O) 
91(z,5) = ç(O9o(Z))= O, 1 + zS' 1 + zS' ' 
and (/9 l(Z, 5) determined by fl and 91 is given by 
| O ( 2 0 VZ(Z--I) 0 --2z  ) 
1 z(1 + z) 0 --z(1 + z) 0 
çl -- (1 + z)  I (z- 1) 0 (z- 1)  0 z(z- 1) , 
-(1 + z) 0 1 + z 0 
k - 2  0 (z - 1) 0 2z 

which is obviously a pseudo-holomorphic curve into G2,5. Similarly, we have 
(252 2v5 2) 
f2=qî(Ofl) = (l+zS) 2' O, (l+z5) 2' O, (l+z2) 2 ' 
.,0 2 = çî(Ogl)= (0, O, O, O, 0), 

and q2, determined by f2 and g2, is given by 

1 

Z252 0 -- V/-Z2 0 Z 2 
0 0 0 0 0 
-vz5 2 0 2z5 0 -vz 
0 0 0 0 0 
52 0 -v'5 0 1 

q2 is an anti-holomorphic curve, which is isonorphic to the Veronese curve, in CP 2. 
Hence we obtain a harmonic sequence from q0: 
0' 0' 0' 0' 
0 --- (/9 = (/:90 --- (/91 "-' :92 ---0. 
By a straightforward computation we obtain 
3 2 
fO -- Il -- 12 = O. 
(1 -t- zS) 2' (1 + zS) 2' 
It is very easy to see that K(ç0) = 4/3, K(çl) = 4/5, K(ç2) = 2 and Al = 3/2. 
It is well known that the rigidity of holomorphic curves in Grassmannians fails; 
so this example is a special harmonic sequence. 

5. PINCHING THEOREM FOR CURVATURE AND KHLER ANGLE 

In this section we will discuss curvature pinching and K£hler angle pinching of 
non-degenerate pseudo-holomorphic spheres in complex Grassmann manifolds. 
Let ç = q« : S 2  Gko,, be a pseudo-holomorphic curve with the non- 
degenerate associated harmonic sequence (5), and let a0 be the length of its asso- 
ciated harmonic sequence. Then from (28) we have 
(41) 5« = --5«_ 2 A- 25«-1 -- 7-«- 1 -- 2 
for a = 1,.-. ,a0, and 
(42) -r« = ((« -- («-bl) -- ((«--1 -- («) -- 2 

for a = O, 1,--- ,ao - 1. 
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It is an immediate consequence of (41) and (42) that 
--I 
(43) 5« ---- (( + 1)(5o -() - 
B--0 
for ( = 1,--- ,(o, and 
(44) fo +--- + r« = (5« - 5«+,) + 5o - 2(( + l) 
for ( -- 0, I,-.- , (o - I, where 5o is the degree of the holomorphic map :o in (5). 
From (43) and (44) we bave a]so 
«o- I 
E (Co - a)-« --(ao+ 1)(5o - ao) 
o--0 

(45) 
aud 
(46) 

&0-- nt_ 1 co-1 
5 = (+ 1)(o-)+ (+ ) +--  (o- )» 
Oo + I o + 1 
B=O 
Dcnoting ç = rein {o,"   , çao- } ( 0), we immediately obtain 
1 
(7) 5o _> o( + )2 , 5 _> ( + 1)(o - )( + 
and "=" holds if and only if ço = "'" = ç«o-, where a = 0, 1,--- , ao - 1. 
Obviously, ç is a totally unramified non-degenerate pseudo-holomorphic minimal 
immersion, i.e., the harmonic sequence ço,"" , ç«o : $2  G, is non-degenerate 
and totally unramified if and only if the degree 5o of ço is ao. For a totally 
unramified non-degenerate harmonic sequence ço,"" , ç«o : $2  G, we bave 
(s) * = (a+ 1)(ao - a). 
At first, by using the Gauss-Bonnet threm we bave 

4 

and - = "r for ail . 
(i) g i«(:) < 

4 
and if - = r for all /, then 
(ao + 2a(ao - a))(1 + -r) 
2 
4 
«(:) = 
1 " 
(Oo + 2(((o - a))(1 + r) 

Lemma 5.1. Suppose that the curvature h of :« satisJïes either Km >_ 
4 4 
or Km <_ 5«_ + 5« Then K« = 5«_ + 5" 
Remark. In Lemma 5.1 we do not need to assume that :« is non-degenerate. 
Theorem 5.2. Let : : S 2 -- G¢,, be a pseudo-holomorphic curve with non- 
degenerate associated harmonic sequence, and suppose that : is the a-th element 
:« of its non-degenerate associated harmonic sequence. 
4 
- (ao + 2a(ao - a))(1 + T) then 
4 
K(:) = 
1  
(ao + 2a(ao - a))(1 + -) 
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Proof. From (47) we see that 
1 
(c-I -[-(c -- (o0-[- 2a(a0- a))(1 + T), 
with equality if and only if T = T for all /. The result is now immediate from 
Lemma 5.1. [] 
_ a(a0-a+l) 
Remark. We have t - (a + 1)(a0- &) under the assumption of Theorem 5.2. 
This shows that the K/ihler angle 0s is independent of T. 
The following is an immediate consequence of Theorem 5.2. 
Corollary 5.3. Let  : S 2 -- Gk,n be a holomorphic curve with non-degenerate 

associated harmonic sequence. 

Then K(ç) - 

o0(1 + 50-) 

Suppose K(ç) <_ 

4 

1 
ao(1 + T) 

and T = T for all . 

Similarly, the following theorem is also an immediate consequence of Theorem 
5.2. 

Corollary 5.4. Let ç  S 2 --, Gk,, be a holomorphic curve with non-degenerate 

associated harmonic sequence, and suppose K(ç) >_ 
4 
1 ' 
a0(l+T) and T = T for all . 

4 

ao(1 + T) 

ThenK(ç) = 

We now prove a pinching theorem for the K/ihler angle. Let 
be a pseudo-holomorphic sphere and let ç0,"" , ço be the associated harmonic 
sequence. We assume that ç = 

Lemma 5.5 ([1]). /f the Kiihler angle t of ça satisfies either t >_ or 
5 
5c-1 5c--1 
t <_ --, then t - 
Lemma 5.6. Let ç be a pseudo-holomorphic curve with non-degenerate associated 
a(ao - a + 1) 
karmonic sequence. If TW= T for all , and t satisfies eitker t  
( + 1)(0 - ) 
(0 -  + 1) the t (o -  + ) 
ot  (+ )(0- )' = (+ )(0- )" 
The proof of the above theorem is immediate from Lemma 5.5 and (47). 
Using (46), we can also prove the following. 
Theorem 5.7. Let ç  S   G, be a pseudo-kolomorpkic curve with non- 
degenerate associated karmonic sequence, and suppose tkat ç is tke a-tk element 
1 &esp. t > 2), 
ç of ifs non-degenerate associated karmonic sequence. If t   _ 
tken t = 0 (resp. t = oe), i.e., ç is a kolomorpkic &esp. anti-kolomorphic) 
crY. 
Proof. When a ¢ 0 and a0, by (46) an immediate computation shows that 
1 -1 
-<<2. 
2  
1 (resp. t > 2), then a 0 (resp. a a0), i.e., ç 
Hence, by Lemma 5.5, if t   _ = = 
is a holomorphic (resp. anti-holomorphic) curve.  
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We believe that -r =p- 0 for the non-degenerate harmonic sequence associated to 
the holomorphic curve of constant curvature, except for the Veronese sequence. 
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THE PERIODIC EULER-BERNOULLI EQUATION 

VASSILIS G. PAPANICOLAOU 

ABSTRACT. We continue the study of the Floquet (spectral) theory of the 
beam equation, namely the fourth-order eigenvalue problem 
[«()"()]" = p()(), - <  < , 
where the functions a and p are periodic and strictly positive. This equation 
models the transverse vibrations of a thin straight (periodic) beam whose 
physical characteristics are described by a and p. Here we develop a theory 
analogous to the theory of the Hill operator -(d/dz) 2 + 
We first review some facts and notions from our previous works, including 
the concept of the pseudospectrum, or p-spectrum. 
Out new analysis begins with a detailed study of the zeros of the function 
F(A; k), for any given "quasimomentum" k E C, where F(A; k) = 0 is the 
Floquet-Bloch variety of the beanl equation (the Hill quantity corresponding 
to F(A; k) is A(A) - 2 cos(kb), where A(A) is the discriminant and b the period 
of q). We show that the multiplicity re(A*) of any zero A* of F(A; k) can be 
one or two and re(A*) = 2 (for some k) if and only if A* is also a zero of 
another entire function D(A), independent of k. Furthermore, we show that 
D(A) has exactly one zero in each gap of the spectrum and two zeros (counting 
multiplicities) in each p-gap. If A* is a double zero of F(A; k), it may happen 
that there is only one Floquet solution with quasimomentum k; thus, there 
are exceptional cases where the algebraic and geometric multiplicities do not 
agree. 
Next we show that if (a,f) is an open p-gap of the pseudospectrum (i.e., 
a < f), then the Floquet matrix T(A) has a specific Jordan anomaly at A = a 
and A = fL 
We then introduce a multipoint (Dirichlet-type) eigenvalue problem which 
is the analogue of the Dirichlet problem for the Hill equation. re denote by 
{/zn }n. the eigenvalues of this multipoint problem and show that {zn }n. is 
also characterized as the set of values of A for which there is a proper Floquet 
solution f(x; A) such that f(0; A) = 0. 
We also show (Theorem 7) that each gap of the L2()-spectrum contains 
exactly one zn and each b-gap of the pseudospectrum contains exactly two 
zn's, counting multiplicities. Here when we say "gap" or "p-gap" we also 
include the endpoints (so that when two consecutive banals or p-bands touch, 
the in-between collapsed gap, or p-gap, is a point). We believe that {znnaz 
can be used to formulate the associated inverse spectral problem. 
As an application of Theorem 7, we show that if t/* is a collapsed ( "closed" ) 
p-gap, then the Floquet matrix T(t/*) is diagonalizable. 
Some of the above results were conjectured in out previous works. However, 
our conjecture that if ail the p-gaps are closed, then the beam operator is the 
square of a second-order (Hill-type) operator, is still open. 

Received by the editors November 13, 2001 and, in revised form, November 10, 2002. 
2000 Mathematics Subject Classification. Primary 34B05, 34B10, 34B30, 34L40, 74B05. 
Key words and phrases. Euler-Bernoulli equation for the vibrating beam, beam operator, Hill 
operator, Floquet spectrum, pseudospectrum, algebraic/geometric multiplicity, multipoint eigen- 
value problem. 

(2003 American Mathematical Society 

3727 



3728 VASSILIS G. PAPANICOLAOU 

1. INTRODUCTION 
The terre "periodic Euler-Bernoulli equation" refers to the eigenvalue problem 
(1) [a(x)u"(x)]"= Ap(x)u(æ), -oe < x < oe, 
where a(x) and p(x) are strictly positive and periodic with a coramon period b, 
satisfying the smoothness conditions a E C 2 () and p E C(). Furthermore, 
without loss of generality, a(x) and p(x) are normalized so that 
(2) La(x)J dx=b. 
One advautage of this normalization is that the ymptotics of certain quantities, 
such  A  oe, become simpler, and this is the only tenon that (2) is used in 
the present work (ste Section 3). 
The study of (1) w initiated by the author in [31] and [33]. There are theoret- 
ical  wcll  practical ferons for studying (1). The Floquet (spectral) theory of 
(1) is ficher than its second-order counterpart (namely the Sturm-Liouville pro 
lem with periodic coefficients, also known  HilFs equation, or the one-dimensional 
SchrSdinger equation with a periodic potential). All the main second-order proper- 
ties continue to hold, while new interesting phenomena arise which are nonexistent 
in the second-order ce. On the practic side, we notice that a typical application 
of (1) is that it models the transverse vibrations of a thin straight beam with pe- 
riodic characteristics (see, e.g., [36] or [17]). Elmtic structures consisting of many 
thin elements arranged periodically are quite common. Although there are some 
authors who have studied such problems numerically (see, for example, [28]),  far 
 we know, [31] and [33] are the only theoretical works on (1). However, recently 
there is an increing interest in higher-order periodic eigenvalue problems (e.g., [2], 
[6], [7]). For results on the second-order inverse periodic problem, or higher-order 
nonperiodic inverse problems, the reader may see, e.g., [3], [4], [5], [8], [10], [14], 
[s], [;], [2], [2], [2si, [2], [2;], [2], [s], [s]. 
The present work continues the investigation on the Floquet spectral theory of 
(1), initiated in [31] and [33]. The goal is a theory analogous to the theory of the 
Hill operator -(d/dx)  + q(x). 
In Section 2 we review some facts and notions from our previous works, including 
the concept of the pseudospectrum, or ¢-spectrum, introduced in [33]. Theorems 
numbered here by capital Latin letters have been proved in out previous works [31] 
and [33]. At the end of Section 2 we have included a subsection containing some 
ide on the significance of the pseudospectrum. 
The new analysis begins in Section 3. We first describe (in Subsection 3.1) a 
technique we use for proving certain theorems (Theorems 2 and 7 of this work). 
Then, in Subsection 3.2 we give a detailed study of the zeros of the function F(A k), 
in the spirit of [16], for any given "quimomentum" k  C, where F(A; k) = 0 is 
the Floquet-Bloch variety of the beam equation (the Hill quantity corresponding 
to F(A; k) is A(A) - 2 cos(kb), where A(A) is the discriminant of the Hill operator 
and b the period of q). We show that the multiplicity re(A*) of any zero A* of 
F(A; k) can be one or two and re(A*) = 2 (for some k) if and only if A* is also a 
zero of another entire function D(A), independent of k. rthermore, we show that 
D(A) h exactly one zero in each gap of the spectrum and two zeros (counting 
multiplicities) in each ¢-gap. If A* is a double zero of F(A; k) it may happen 
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that there is only one Floquet solution with quasimomentum k; thus, there are 
exceptional cases where the algebraic and geometric multiplicities do not agree. 
In Section 4 we first (Subsection 4.1) review briefly some facts regarding certain 
operators La, where k E C is the quasimomentum, and then, in Subsection 4.2, we 
apply them to show that if (c,/3) is an open ¢-gap of the pseudospectrum (i.e., 
c </3), then the Floquet matrix T(A) has a specific Jordan anomaly at A = c and 
A =/3 (this was conjectured in [31] and [33]). 
In Section 5 we introduce a multipoint (Dirichlet-type) eigenvalue problem which 
is the analogue of the Dirichlet problem for the Hill equation. We denote by 
the eigenvalues of this multipoint problem and show that {#},ez is also charac- 
terized as the set of values of A for which there is a proper Floquet solution f(x; 
such that f(0; A) = 0. If we normalize f so that f(0; A) -- 1, then {#,},ez is the 
set of poles of f(x; A) (viewed as a function of A, of course) counting multiplicities 
(this approach is used in [11]). 
We also show (Theorem 7) that each gap of the L2()-spectrum contains exactly 
one Pn and each ¢-gap of the pseudospectrum contains exactly two pn's, counting 
multiplicities. Here when we say "gap" or "¢-gap" we also include the endpoints 
(so that when two consecutive bands or ¢-bands touch, the in-between collapsed 
gap, or ¢-gap, is a point). We believe that {#,},ez can be used to formulate the 
associated inverse spectral problem. 
As an application of Theorem 7, we show that if u* is a collapsed ("closed") 
¢-gap, then the Floquet matrix T(u*) is diagonalizable (this too was conjectured 
in [31] and [33]). 
Some of the above results were conjectured in our previous works. However, 
the formulation of the inverse problem (and, in particular, our conjecture that if 
all the ¢-gaps are closed, then the beam operator is the square of a second-order, 
Hill-type, operator) remains open. 

2. IEVIEW OF EARLIER RESULTS 
2.1. The spectrum. We start by recalling certain general facts related to (1) (see 
[13], Sec. XIII.7) and some of the main results established in [31] and [33] (other 
references for Floquet or periodic spectral theory are, e.g., [9], [10], [16], [18], [19], 
[22], [34]). The problem is selfadjoint (with no boundary conditions at -t-oe). The 
underlying operator L (the "Euler-Bernoulli operator" or "beam operator" ) is given 
by 
Lu = p- (au")". 
The corresponding Hilbert space is the p-weighted space Lp 2 (1). Notice that L 
is a product of two second-order differential operators, namely L = L2L, where 
Lu = -au" and L2u = -p-u'. 
For any fixed  the "shift" transformation 
(Tu)(x) = u(x + b) 
maps solutions of (1) to solutions. As a basis of this space we take the solutions 
uj(x; A), j = 1, 2, 3, 4, such that (primes refer to derivatives with respect to x; 5jk 
is the Kronecker delta) 
(k-1)(O;) =(jk, k= 1,2, a(O)lt(O;) =(j3 ` [alt]' (0;))=(j4- 
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We refer to uj as the j-th flndamental solution. Each uj(x; A) is entire in A of 
order 1/4. We identify T with its matrix representation with respect to the above 

basis (called Floquet matrix), namely 

l(b) 2(b) u3(b) 4(b) 
; (b) «(b) u'(b) ',(b) 
a(b)uï(b) a(b)u(b) a(b)u(b) a(b)u(b) 
[auï]' (b) [au'']' (b) [aug]' (b) [au'l' (b) 

where the dependence in A is suppressed for typographical convenience. In [31] it 
was shown that the eigenvalues rl,r2,r3,r4 of T (called Floquet multipliers) appear 
in pairs of inverses, namely 
(3) rit4 = r2r3 = 1 
(in fact this is truc for any selfadjoint ordinary differential operator with real, 
periodic coefiïcients). It follows that the characteristic equation of T has the form 
(4) r 4 - A(A)r 3 +/(A)r 2 - A(A)r + 1 = O. 
Exccpt for a discrcte sct of 's, T = T(A) is similar to a diagonM matrix d 
its eigenvectors correspond to the Floquet solutions, namely to the solutio fj, 
j = 1.2,3,4 of (1) such that 
(5) fj(z +b)=rjfj(x). 
There are four linearly independent Floquet solutions if d only if T is similar to 
a diagonal matrix. We also notice that (5) implies 
fj(x) = eWXpj(x), where ri = e w'b and pj(x + b) = pi(x). 
The L ()-spectrum S(a, p) of (1) can be characterized as the set 
(6) S(a,p) = {A  c: I(A)I = 1, for some j}. 
It c be shown that S(a, p) is rem with inf S(a, p) = 0. In the unperturbed case, 
i.e., when a(x)  p(x)  1, we have 
S0 f S(1, 1)= [0, oe) 
(the index 0 indicates that a quantity belongs to the unperturbed case). 
Next, for a fixed real number k, we consider the corresponding k-Floquet eigen- 
value problem on (0, b), nely 
(7) [a(x)u"(x)]"= Ap(x)u(x), u(J)(b) =ekbu(J)(O), j =0,1,2,3. 
k 
Let {A( )}= be the spectrum of (7). Since the problem is selfadjoint, A(k)  
. The eigenvalues can be indexed so that A(k)  A+l(k). Then An(k)  
Ch . We also notice that, since A(k + 2/b) = A(k), one only needs to consider 
k in [0,2/b). Furthermore, (3) implies that A (2/b-k) = A(k). The set 
{A_, «er A(0)}= is the periodic spectrum, while {A ce, A(/b>}  
= = is the 
tiperiodic spectrum. The n-th bd of S(a, p) is the closed interval 
= U () 
O<k</5 
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and it is well known that S(a,p) = J,=l B,. In [31] it was shown that (as in the 
Hill equation) 
B2m+l ["2m, ! , !  .... 
= "2m÷l] B2m÷2 = [,'2m÷2,,'2m÷l] , m 0,1,2, 
In fact, as A moves, say with constant velocity, from A2m to A,+ 1 (resp. from 
,k,+2 to A,+I) two Floquet multipliers, say r and r3 = -, move sinoothly on 
the unit circle, with nonvanishing speed (i.e., without changing direction), starting 
at 1 and arriving at -1 (resp. starting at -1 and arriving at 1). The other two 
multipliers, ri and ?'4, stay real. In particular, A,(k) is strictly monotone in k, 
on [0, tf bi; hence the interior of B, is never empty. An interesting question here 
(pointed out by the auonymous referee) is whether we always have [dAn/dk[ > 1, 
as in the Hill case (see [19]). 
Two bands can "touch" each other (when A,+I = A,+2 or A,+ 1 = A,+), 
but they cannot overlap (i.e., they have disjoint interiors). The gaps of the spectrum 
S(a, p) are 
! ! 
I2m-1 = (,2m_l,,2m) , I2m = (,2m_l,,2m), Tf/ = 1,2,3,..., 
and empty gaps are traditionally called "closed". If ,'2m--1 ( "2m (resp. ,k,_ 1 < 
,), then r (and ra) has square root branch points at ,k = ,k,_,, ,k, (resp. at 
A = A2,_ 1,' A,).' If, on the other hand, A,-I -- A, (resp. A,_ 1 --- A,),' i.e., 
if the corresponding gap is closed, then r2 (and r3) are analytic about ,k = ,k2m_ 1 
! 
(resp. about ,k = ,k2m_ 1). The value ,k = A0 = 0 is very special since ail Floquet 
multipliers have a fourth foot branch point there and T has only one eigenvector. 
If   0, then the characteristic equation of T can only bave siinple or double 
roots. Now let A g: 0 be such that (4) has a double root, say ri. Then there is one 
Floquet solution fj(x + b) = rfj(x) and a solution g(x) (f and g are linearly 
independent) such that gj(x + b) = rjgj(x) + cjfj(x), where the constant ci may 
be 0 (in this case we say that we have coexistence, i.e., two linearly independent 
Floquet solutions corresponding to the saine multiplier). If ci -J= 0, we can say that, 
for this particular A, T has a Jordan anomaly (this teriniilology is due to Professor 
Barry Simon) and that gj (x) is a generalized Floquet solution of (1). 
We now review the main results of [33]. Notice that, in that reference it was as- 
sumed that a, p E C 4 (IR), but we believe that they remain true for a E C = (IR) and 
p e C (IR), and here is why: All these results concern entire functions of A that are 
polynomial expressions of the uj(b; A)'s. By considering a(x) and p(x) as limits of 
smooth (C 4 or even C m) functions an(x), pn(X) ill the C 2- and sup-norms respec- 
tively, the corresponding entire function uj,, (b; A) converges to uj (b; A), uniformly 
on compact subsets of C (see the proposition in the Appendix), for j = 1, 2, 3, 4. 
Thus, we think that our results extend immediately to less smooth coeflïcients. 
For a fixed real number k, equation (4) implies (by setting r = e ikb) that the 
k-Floquet eigenvalues of (7) are the zeros of the entire function 
(8) F(A; k) = B(A) - 2A(A)cos (kb) + 4cos  (kb), 
where we have set 
(9) B(A) =/(A) - 2. 
In the unperturbed case, this function becomes 
(10) F0(A; k)=4 [cosh (/l/4b) -cos (kb)] [cos (/l/4b) -cos (ïb)] . 
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This expression implies easily that, if 0 < k < n/b, then the zeros of F0(A; k) are 
(11) An,0 = 2 In/2] r/b- (-1)nk , n = 1,2,3 .... 
(where [-] is the greatest integer function) and they are all simple. Purthermore, 
An,0 lies in the interior of the n-th band Bn,o, for every n. 
(Theorems that have been proved in previous articles are numbered by upper- 
case Roman letters.) 
Theorem A. Let 0 < k < r/b. Then the zeros {)n(k)}n=l of F(A;k) are simple, 
An(k)  Bn, and fo each An(k) corresponds a unique eigenfunction 0n(x; k) of (7), 
i.e., the geometric multiplicity of An(k) is also 1. 
Theorem B. The multiplicity of any zero of F+(A) de__f F(A;0) can be only 1 or 
2. A zero A* of F+(A) is double if and only if A* = A2,- = A2,, for some 
m >_ 1 (i.e., the corresponding gap is closed). Furthermore, (7) bas two (linearly 
independent) periodic eigenfunctions corresponding fo A* (coexistence) if and only 
if A* -- A2,-_ -- A2,-. Thus, we tan say that the algebraic multiplicity of any 
periodic eigenvalue is equal fo its geometric multiplicity. The same things also hold 
for F-(A) de=_f F(A;r/b), which is the entire function associated to the antiperiodic 
case. 
2.2. The pseudospectrum. The previous results are the exact analogues of the 
results for the Hill equation regarding algebraic and geometric multiplicities (see 
[161). 
In [33] we introduced a concept that, as far as we know, does hOt have a coun- 
terpart in the second-ortier case: 
Definition. Let k  (0, /b). The set 
k (a, p) ---- {A  C" there are two Floquet multipliers ri(A ), r(A) of (1) 
such that ri =  = Irai e kb, ]rj] > 1} 
is called the k-Floquet pseudospectrum (or k-Floquet ¢-spectrum) of (1). We, 
furthermore, call the set 
(')= U (,) 
O<k<w/b 
the pseudospectrum (¢-spectrum) of (1) on the line (here D denotes the topological 
closure of D). 
The following entire function was introduced in [33]: 
(12) G(A; k) = A(A) 2 - 4B(A) cos 2 (kb) - 16 cos 2 (kb) sin 2 (kb). 
It follows that, if u  k(a,p), then G(u; k) = 0. 
Since 
G(A; k)=G(A;r/b-k), 
the zeros of G(A; k) also include ,/b- (a, p). 
Conversely, for a fixed k  (0, r/2b), let  be a zero of G(A; k). Then (12) implies 
that 
(13) A(,)  - 4B(,) cos 2 (kb) - 16 cos 2 (kb) sin  (kb) = O. 
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Now by (3), (4), and (9), 
(14) A - T 1 ÷ -- ÷ T 2 ÷ -- and B = T 1 ÷ r 2 ÷ -- ; 
Tl 72 72 
hence, given rl, (13) becomes a 4-th degree (algebraic) equation in r2. One can 
check that its solutions are 
T 2  T1 e-t-2ikb and r2  Ïle -t-2ikb. 
This means that 
P G k (a,p) U r/b-k Ca, p) = {A: ri = r,e +2ikb, I1 ¢ 1}. 
Therefore, for k  (0, r/b), k ¢ r/(2b), the set of zeros of G(A; k) is k (a,p) U 
rlb-k (a, p). 
The value k = r/(2b) is somehow special since 
(15) G(A;rr/(2b)) = A(A) 2. 
It follows that, if P is a zero of G(A; 7r/(2b)), thon 
r2 (P) ---- --ri (P) or r2 ---- --ri (p)-l . 
Hence 
(notice that, in this case all Floquet multipliers are pure imaginary). These results 
are implicitly contained in [33], where the following theorem was established: 
Theorem C. Let 0 < k < 7r/b. If k ¢ 7r/(2b) the zeros of the entire func- 
tion G(A; k), defined in (12), are all real, strictly negative, and simple. The ze- 
ros of G(A; rr/(2b)) are ail real strictly negative and double. Each zero Pn (k) of 
G(A; k) is (as a function of k} strictly monotone on the interval (0, rr/(2b)) and on 
(r/(2b),r/b). 
The function G(A: k), as defined in (12), makes sense for all k  C (in fact, it is 
entire in A, k). In particular, 
(16) E(A) «ef G(A;0)= A(A)2 - 4B(A). 
This function was introduced in [31]. It was shown there that 0 is always a simple 
zero of E(A) and that P = 0 is a zero of E(A) if and only if ri(P) = ri(P) ¢ +1, 
j = l. In [33] the following theorem was proved: 
Theorem D. The nonzero zeros of E(A) are all real, strictly negative, and simple 
or double. If we denote them by 
o = Po > Pi > P > P1 > P > P > P' >..., 
we have a pseudoband-pseudogap structure on the negative A-axis. Each ¢-band 
[oe0, oe], [oe,oel], [oe2, oe] ,--- contains exactly one point of the ¢-spectrum k (a, p), 
fo anu ed k  (0, /b). 
Remark 1. Since 0 is always a simple zero of E(A) and in the unperturbed case we 
have 
[ ( ) ( )] 
(17) E0 (A) = 4 cosh AlAab - cos AlAab , 
it fol]ows by Theorem D and a simple continuity argument that E(A) > 0, if A > 0 
or if A is in the interior of a ¢-gap; whereas E(A) < 0, if A is in the interior of a 
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! 
¢-band. At the zeros un, u n, n -¢ 0, of E(X), there are Floquet multipliers ri, 
j ¢ l, such that r (u) = rt (u) > 1 and ri (u) = rt (u) < -1. 
If a(x)p(x)  1 (iu this case the beam operator is a "perfect square"), then all 
the nonzero zeros of E(A) are double, i.e., AE(A) is the square of an entire function. 
Equivalently, all ,¢-gaps are closed, i.e., empty. 
For the unperturbed case we have 
The zeros of G0(A; k) are u,o(k) and u,,o(/b - k), n = 1,2, 3, ..., where 
u,,o(k) = -4[2 Ln/2J /b - (-1)"k] 4 
Next we set 
u-,0 u,0 (0) lira u.o(k), ' 
= = un, o = un,o(/b) = lira u,.o(k). 
ko 

Thus, Vo,o = 0 and, for m = 1, 2, 3 .... , 

(18) 

l]2rn_l, 0  l]2rn, 0  --4 

l2m_l, 0 -- l2rn, 0 -- 

These numbers are the zeros of E0 (X). 

(), 
(m - 1) 

2.2.1. The significance of the pseudospectrura. The purpose of this short (sub)sub- 
section is to elucidate certain things regarding the concept of the pseudospeetrum. 
Let L be an n-th order (ordinary) differential operator with periodic coefficient,s. 
Then one tan consider the Floquet multipliers rj(X), j -= 1 ..... n, of L and the 
corresponding Floquet solutions fj(x;X), j = 1 .... ,n. The rj(X)'s are, in fact, 
the branches of a (multivalued) analytic function which we denote by r(X) and, 
similarly, the fj(x; A)'s are the branches of a A-analytic function f(x; A). 
Let F be the Riemann surface of r(A). If we nornmlize f(x; A) so that f(0; A) : 1, 
then f(x; A) becomes a meromorphic function on F, whose set of poles we denote 
by 
In [11] it is suggested that the (periodic) inverse spectral data for L is the Rie- 
mann surface F together with the set of poles {#,} (notice that each #, is a point, 
on F, i.e., #, is not just a complex number, since it also contains the information: 
on which sheet of F the pole lies). This is, of course, inspired by the inverse theory 
of the Hill operator (sec, e.g., [11], [14], [24], [25], [38]). 
If L is the Euler-Bernoulli operator, the multiplier r(X) has two types of branch 
points (the point, X - 0 is special and tan be considered as being of both types). The 
braneh points of the first type lie on the positive real axis and are the endpoints 
of the bands of the L()-spectrum (they are also the periodic and antiperiodic 
eigenvalues), exactly as in the Hill case. The branch points of the second type lie 
on the negative real axis and they, too, define a band-gap structure which we bave 
called pseudospectrura (we have called its bands and gaps "b-bands" and "b-gaps" 
respectively to distinguish them from the spectral bands and gaps). 
Each gap of the speetrum eontains exactly one #,. But, now there are #n's that 
do not lie in any spectral gap. It turns out that each b-gap of the pseudospeetrum 
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contains exactly two of those #ri's, counting multiplicities. The exact statement is 
Theorem 7 of Section 5, which is, perhaps, the crux of this work. 
In conclusion, for the Euler-Bernoulli case, we need both the Lp2(lR)-spectrum 
and the pseudospectrum in order to determine the Riemann surface F 
and the intervals in which the pn's are confined. This is why we believe that 
the pseudospectrum plays an essential role in the Euler-Bernoulli inverse spectral 
theory. In particular, we have conjectured that if the pseudospectrum has no gaps 
(i.e., if it is the interval (-oe, 0]), then the Euler-Bernoulli operator is a perfect 
square of a Hill-type operator. 

3. THE ZEROS OF THE FUNCTION F(A; k), FOR COMPLEX k 
3.1. The technique. There is a technique that we bave employed for proving some 
of our statements regarding (1), especially properties of quant ities t hat depend on 
(or are related to) the spectral parameter A (e.g., this technique has been already 
used for proving Theorems C and D mentioned above). It combines contilmitv 
arguments and large lAI asymptotics. 
Here is how the technique works: We first check that the property we want to 
establish holds in the unperturbed case a(x) = p(x) = l. Then we deform a(x) and 
p(x) continuously until we reach the geueral case, making sure that the property 
remains valid (a kind of "continuous induction"). For example, we can specify the 
(obviously continuous) deformation 
(19) 
ct(x;t)=tct(x)+(1-t), p(x;t)=«(t)4[tp(x)+(1-t)], t  [0,1], 

where 

c(t) = 

b 

fô [tp(x)+(1-t)] 1/4 
ta(x)+(1-t)J 

dx 

Notice that 
a(x;O)=_p(x;O)=l, a(x;1)=a(x) and p(x:l)=p('). 
Also, since a(x) and p(x) are strictly positive, it follows that there is a constant 5o, 
independent of t, such that 
a(x; t), p(x; t) >_ 50 > O, for ail t  [0, 1], 
where a(x; t) and p(x; t) satis .fy the normalization condition (2), namely 
f0  [P(;t)] 
[a(x;t)] dz=b, forall t[0,1]. 
Finally, as functions of x, a(x; t) and p(x; t) are b-periodic and as smooth as a(x) and 
p(x) respectively. 
In some cases, we first prove the desired result for a(x) and p(x) that are suf- 
ficiently smooth, say a, p  C 4 (ll), and then extend it to the more general case 
a  C 2 (]R) and p  C (]R) by approximating a(x) and p(x) by smooth functions (in 
the C 2- and sup-norms respectively). The fact that a A-quantity of the smooth case 
approaches (uniformly on compact subsets of C) the corresponding A-quantity of 
the more general case usually follows from standard Gronwall-type estimates (e.g., 
sec the proposition in the Appendix). 
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One main reason that we need smooth a(x) and p(x) is that, if this is the case, 
there is a Liouville-type transformation (round in [3]) that transforms (1) to a 
canonical fourth-order eigenvalue equation, nmnely 
v'"' (4) - [ql ()v' ()1' + q2 ()  () = 

(20) 
where 

 = [-T] dy, () = p()3/Sa(x)I/s(), 
and q (ç), q2() are periodic expressions involving  and p (see [3]). 
Then one tan use in (20) the asymptotic estimates of [29], Part I, Chapt. Il, to 
conclude that, in each sector 
Si={ AC'l<arg(A)< (1+1)}4 - - 4 , /=0,1 .... ,7, 
of the complex A-plane there are four A-analytic linearly independent solutio 
¢j(x;t;A), j = 1,2,3,4, of 
[(x;t)"(x)]"= x(; t)(), 

(21) 
such that, given /1 > O, 
(2) 
¢ (:: t; ,) - 

(23) 

(24) 
(25) 
where 

eeJ)d/aS(x;t) I 
p(x; t)3/8a(x; t)l/8 <- 

t; ,) - 
p(x; t)/Sa(x; t)3/s 
I 

p(3:;t) 1/8 2 1/2 e,'/aS(x;t) 
a(x;t)SlS% e 

l e: j ) 1 I« S(x;t) I 
K O<_x<_M, 
11/4 ' 

a(x;t)7/se'\a/4e''l's(=;t) <- K A/2e''l's(=:t) , 0 <_ x < M, 

S(x;t) = oo  [P(Y;t)] 114 
L/ d 
(in particular, S(nb; t) = nb, if n  Z). Here A 1/4 stands for the principal branch of 
the fourth root (so that .{A 1/4} >_ 0, {A /4} >_ 0), {¢,¢2,e3,¢4} = {i,-1,-i, 1}, 
and the (positive) constant K depends on a(x), p(x), and M, but hot on t. 
We finish this subsection with a useful lemma. 

Lemma 1. If r (A;t), j = 1,2,3,4. are the Floquet multipliers of (21), where 
t  [0, 1], then 
(26) ri (A;t) K 
e/b 1 <_ ]A[1/, j = 1,2,3,4, 
where A U4 is the principal branch of the fourth root, {al, e2, e3, e4 } = {i,- 1,-i, 1 }, 
and the constant K > 0 depends on a(x), p(x), but not on t. 
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Proof. Let 
T = [Tjk]l<_j,k<_4 

be the Floquet matrix of (21) with respect to the basis Ci(x; t; A), j = l, 2,3,4, 
where Ci(x; t; A) is the solution of (21) that satisfies (22), (23), (24), and (25). It 
follows that, as lA I -- o% 
where 5jk is the Kronecker delta. As we know, ri (A; t), j = 1,2, 3, 4, are the roots 
of the equation 
r 4 - A(A;t)r 3 + [B(A:t) + 2If 2 - A(A:t)r + 1 : O, 

4 
A(A; t)= 
j----1 

where 

and 

B(A; t)+ 2 = Z 
l_j<k_4 

Thus (27) implies that, as lAI --} oc, 

and 

uniformly in t. The statement of the lemma follows easily from these two formulas. 

3.2. The theorems. The analysis that follows is inspired by the work [20] of W. 
Kohn for the second-order case (see also [1]). Consider again the fimction of formula 
(8), namely 
F(A; k) = B(A) - 2A(A)cos(kb) + 4cos 2 (kb). 
This function is the analogue of the fimction 
A(A)- 2cos (kb) 
that appears in the analysis of the Hill operator (see, e.g., [1]). Notice that, although 
F(A; k) is usually viewed as an entire function of A, it is entire in both A and k. We 
want to generalize Theorems A and B for the case of complex k. 
Theorem 1. For a fixed k E C, let A* be a zero ofF(A; k) ofmultiplicity m(A*) > 1. 
Then A* is also a zero of the entire function 
(28) D (A) = E' (A) 2 - 4A' (A) 2 E (A). 

Furthermore, D ( A )  O. 
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Proof. The derivative of F(A; k) with respect to A satisfies 
(29) F(A*; k) = B'(A*)- 2A'(A*)cos(kb) =0. 
A straightforward calculation (using (16)) can verify t, hat 
(30) 
A'(A)2F(A;k) - A(A)A'(A)F(A;k) + 2B'(A)F(A;k) - F(A: k) 2 - D(A) 
16 ' 
where D(A) is given by (28). Thus, if F(A*;k) = F(A*; k) =0, then D(A*) =0. 
If D (A) =_ 0, then all zeros of F(A; k) should have been multiple. But, by 
Theorem A the zeros of F(A; k) are simple, if k is real and 0 < k < 7r/b. Thus, 
D(A)  0. [] 
One important feature of Theorem 1 is that D () is independent of k. The next 
theorem is quite informative. To prove it we employ the technique described in the 
previous subsection. 
Theorem 2. All the zeros of the entire function D () of (28) are real and they 
are located as follows: (a) D () bas exactly one (simple) zero in each gap of the 
spectrum S(a,p) of (1) (with the understanding that, if the gap is closed, i.e., 
collapses to a double periodic or antiperiodic eigenvalue, say *, then the simple 
zero of D () is *); (b) D () bas exactly two zeros (counting multiplicities) in, each 
¢-gap of the pseudospectrum of (1). In case (b), if the ¢-gap is open, then D () 
has exactly two simple zeros in it, whereas if the ¢-gap is closed, i.e., collapses to 
a point u*, where u* = u2,- = u2,, or u* = u,_ 1 = u,, for some n = 1, 2, 3, ... 
(see the statement of Theorem D), then u* is a double zero of D (). There are no 
other zeros of D (  ). 
Proof. We start by examining the zeros of D () in the ¢-gaps. Let ((, ff) be a ¢- 
gap. If it is closed, i.e., if ( =/, then, by Theorem D we have E(() = E'(() = 0. 
Thus (28) implies that D(() = D'(a) = 0 (i.e., ( is a zero of D () of multiplicity 
> 2). Next assume that a </. We then bave E(a) = E(/) = 0, while, by Remark 
1, E() > 0, for   ((,/). It follows that there is a 3   ((,/) such that E'(3) = 0. 
Then (28) implies that 
D(a) = E'(a)  > 0, D(/) = E'(/) 2 > 0, D(') = -4A' (-)2 E(-) < 0. 
Therefore, D (A) lnust have a zero in (a,-) and one in (% ). In conclusion, D (A) 
always has at least two zeros (counting multiplicities) in each ¢-gap, even if it is 
closed. 
Next we consider the unperturbed case. Using E0 (A) of (17), namely 

and the fact that 

./)-- [os (;0)- os (;0)]  , 

o():  [,os(»)+os 
we get from (28) that 
(31) 
16b 2 
Do (A) - A3/ 

sinh()l/4b)sin(,X1/4b) [cosh()l/4b) -cos(,X1/4b)] 



THE PERIODIC EULER-BERNOULLI EQUATION 3739 

(in particular, D0(0) = 16b s # 0). Hence the zeros of D0 (A) are exactly as the 
theorem describes them. As we deform the unperturbed case continuously, until 
we reach (1), D(A) will continue to have two zeros in each b-gap, unless some new 
nonreal zeros enter the b-gap. Also, by Theorem A and Theorem 1, the simple 
zero that D(A) has in each gap of the spectrum S(a,p) (including the case of a 
closed gap) cannot move into the interior of a band. Hence this zero will remain 
in the gap until some other nonreal zero enters the gap. Therefore, as we deform 
the unperturbed case, D(A) will continue to have exactly one zero in each gap 
and exactly two zeros (counting multiplicities) in each b-gap, until solne nonreal 
zero(s) enters a gap or a b-gap. But where can these nonreal zeros corne from? 
Since D0 (A) has no othcr zeros, they can only colne from infilfity. 
From the above it follows that, in order to finish the proof, we need to demon- 
strate that, as we deform the unperturbed case, no new zeros of D(A) can appear 
from infinity. Assume first that a, p E C  (ll). Let D(A*) = 0 where A* does hOt 
satisfy the statement of the theorem. Consider the continuous deformation (19) and 
let D(A; t) be the function D(A) for the probleln (21). Then there is a t0 E [0, 1) 
and a zero A,(t) of D(A; t) such that 

(32) A.(1) = A* and lira IA(t)l = oe 
t\to 

(Au(t) depends continuously on t). 
By Theorem 1 there is a k = k(t)  C such that 

(33) 

F(),(t);k;t) =0 

and 

(34) FA (),(t);k;t) =0. 
An equivalent way to state (33) is to sa.v that 
for some 

ri = ri ((t); t) = e , 
namely 
(35) 
r+r4=r+r - =2cos(kb) 

j=1,2,3,4, 

re + r3 = re + r  

or 
Using (29) and (35), formula (34) becomes 
(36) 
B ()(t);k;t) = Ah ()(t);k;t) [rfi (A(t); t) + rj (A(t);t)-] , 
By recalling (14), (36) becomes 
[ /+/ -] 0 [ 

= 2cos(kb). 

j=lor2. 
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ifj = 1, or 

['1 (/w)÷ '1 (/w) -1 ] O,k ['2 ÷ .-1] ,k=,ko 
: [r2(Aw)÷r2(A) -1] 8,X [r2÷r-l]l=, 

if j = 2. Therefore, 
(37) 
or 

ri (Aw) ÷ '1 (w) -1 - '2 (u) ÷ '2 (w) -1 

[1 -- T 1 (/w) -2] 0,kr 1 ()w) = 0 
OF 
An equivalent way to write the last two equations is 
[l(/w)2-- 1] [2(/u)2-- 1] -0 

(38) 
or 
(39) 

[Arl (/w)] [,X'/2 ()w)] = O. 
Thus, if D(Au; t) = 0, then Au satisfies (37), or (38), or (39). But (37) means that 
E(Au; t) : 0 and, by Theorem 1, this can happen if and only if Au is a double zero 
of E(A; t). Similarly, (38) means that Au is a periodic or antiperiodic eigenvalue 
of (21) and this can happen (see Theorem B) if and only if Au is a double such 
eigenvalue, equivalently if Au is a double zero of F + (A; t)F- (A; t). Therefore, either 
Au (t) is a double zero of 
F+(A;t)F-(A;t)E(A;t), 
or Au(t) satisfies (39). 
Next, let {z_,(t)}= o be the zeros (counting nmltiplicities) of E(A;t) and 
{z,(t)}= o be the zeros (counting nmltiplicities) of F+(A; t)F-(A; t) (thus zo(t) = 
0). Furthermore, we assume that {z,(t)}=_oe is increasing in n. "vVe then have 
the estimates (see [33]) 
(40) Iz(t) -- z(0)l <_ t£n 2, for ail n G Z, 
where the (positive) constant/x" is independent of t G [0.1], z0(0) = 0, and, by (11) 
and (18), 
(4) 
751--21 (0) -- Z--21 (0) -- --4 Z21--  (0) = Z2t (0) = 1 = 1, 2.3,.... 
In particular, there is a constant C > 0 such that 
(42) z,+2(0)- z,(0) > C Inl 3 , for ail n G Z. 
Finally, recall that z,(t) is a branch point of r(A; t) if and only if z,(t) is a simple 
zero of F+(,X:t)F-(,X;t) or 
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We will now show that (32) is impossible. This will imply that there is no A*, 
such that D(A*) = 0, violating the statement of the theorem. Let F = F(A(t), R), 
the circle in the complex plane with radius R, centered at A(t). We assume that 
R is small enough so that F does not enclose any branch point of r(A; t), i.e., any 
(zn(t) which is a) simple zero of F+(A;t)F-(A;t) or E(A;t). Then by Cauchy's 
integral formula we have 
eb )14 b _ 1  ri (z) l e 
() 
Thus, if (t) satisfies (39), 
(43) 
4A/4 -- 2,R rJ(z(8))--eeZ(°)'/%, d8`l 
If we sume in ccordmce with (32) 
d te 
  I()1(/)«, fo om   (0 /4)  
then (43) and (26) impl that there i a contant " independent of  uch t.hat 
aw(t) 3/4 ee  R I(t)l'/4 -- la(t)l(a/4) + 
This inequality is obviously impoesible  IA(t)l gets arbitrarily large; hence 
must encloee branch points. We have, therefore, established the following: Given 
  (0, 1/4), there is a constant A, independent of t, such that, if I(t)l  A, then 
I(t) - z(t)l  l(t)l ('/)+ , o om¢  e z. 
Using (40) and (41) the above estimate can be written  
(44) IA(t) - z(0)l  Klnl u+4 , for some n  Z. 
Finally, since   (0, 1/4), the estimates (42) and (44) make it impoible for 
to move continuously to infinity. Hence (32) is impoible and the theorem is proved 
for a, p  C  (). The general ce follows by approximation by C  functions. We 
just have to observe that D(A) can be written in terres of the fundamental solutions 
uj(b; A) (and their derivatives) and apply the propoeition of the appendix. 
Remark 2. If D(A*) = 0, then the threm implies that A* is in a gap or a ¢gap. 
Thus, the corresponding Floquet multipliers ri(A*), j = 1, 2, 3, 4, are all real (thus, 
ri(A* ) = eikwhere (kj} = 0 or r/b). As ve have already seen (see also Lemnm 
4 below), if A* is in a spectral gap, then we always bave r(A*),r4(A*) > 0, while 
r2(A*), r3(A*) may be poeitive or negative; however, if A* is in a ¢gap, then all 
ri(A*), j = 1,2,3,4, have the same sign. rthermore, if A* is in a gap or a ¢gap, 
then (39) implies that r(A*) = 0, for some j = 1, 2, 3, 4, and conversely. 
The next theorem goes a little deeper. It completes Theorem 1. 
Theorem 3. For a fixed k  C the multiplicity m of any zero A* of F(A; k) can 
 either one or two (of course, by Theorem 1,  = 2 if and only if D (A*) = 0). 
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Pro@ Let A* be a zero of F(A;k) with multiplicity m > 3. Then (30) implies 
that A* is a zero of D (A) of multiplicity al least m - 1. But, by Theorem 2, the 
multiplicity of auy zeros of D (A) can be al most two. Thus, m --- 3 and A* is 
a double zero of D (A). But then, again by Theorem 2, we must have that 
corresponds to a closed b-gap, i.e., il is a double zero of E(A); thus E (A*) = 
E' (A*) = 0. Next we observe that, using (16), we can write (30) as 
A,(A)2F(A:k) E'(A) 
2 F(:k)- F(A;k) 2 = D(A__) 
 16 ' 
which implies that A* is a zero of D (A) of multiplicity 3, a contradiction! Thus, 
F(A; k) calmot have any zeros of nmltiplicity la.rger than two. [] 
\,k' contimm with a lenmm which is by itself interesting since il characterizes the 
zeros of the entire fulctions A(A) and B(A). 
Lemma 2. (a) The set of zeros of A(A) of (4) is the (rr/(2b))-Floquet b-spectrum 
O2rr/2b(a, 13). Furtherrnore, ail zeros of A(A) are simple. 
(b) The set of zeros of B(A) of equality (9) is the (rr/(2b))-Floquet spectrum 
(A,(rr/(2b))},__l. Again. ail z«ros of B(A) are simple. 
Proof. Part (a) follows illmmdiately from (15) and Theorem C. 
Part (b) also follows immediatcly fronl the fact (sec (14)) that B(A) --- 
(ri "}- r-Ç1)(r2 "l- r-l). [] 
Remark 3. The lemma implies that B(A) has one zero (counting multiplicities) in 
the interior of each band of the spectrum of (1) and no other zeros. In particular, 
all zeros of B(A) are (real and) strictly positive aud simple. 
Likewise A(A) has one zero (counting multiplicities) in the interior of each 
band of the b-spectrum of (1) and no other zeros. In particular, all zeros of A(A) 
are strictly negative and simple. 
From Theorems 1, 2, and 3, il follows that, if A* is a lnultiple zero of F(A; k), 
then A*  ]R, ils multiplicity is two, and we must have k  (17 with {k} = 0 or 
r/b (without loss of generality). Using Lemlna 1 we can, in addition, prove the 
following: 
Theorem 4. A gwen number A* can be a double zero of F(A: k) for al most one 
value of k with N{/,-} 

Pro@ Assume 

Then (29) holds, namely 

iv(*; )= F(A*; )=0. 

B'(A*) - 2A'(M)cos(kb) = O. 
If there are two distinct kl -J= k2 with .{kt}, .{k2}  {0, rr/b}, for which the above 
is truc, we must have 
(45) B'(A*) = A'(A*) = 0. 
Now the functions A(A) and B(A) are entire of order 1/4. Since the zeros of A(A) 
are positive and the zeros of B(A) are negative (sec Remark 3 above), il follows by 
a well-known theorem of complex analysis (sec, e.g., [37]) that the zeros of A'(A) 
are positive while the zeros of B'(A) are negative. Thus, (45) is impossible. [] 
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Remark 4. Let D(A*) = 0 where A* is in an open gap or ¢-gap. Then by Theorems 
1, 2, 3, and 4 there is a unique k E C \ ll with .{k} = 0 or /b such that A* is 
a double zero of F(A; k). The four Floquet multipliers that correspond to A* are 
distinct (since A* lies in an open gap or ¢-gap) and one of them is r = e Zkb (where 
r E ll, r = +l). Hence there is only one (up to linear independence, of course) 
Floquet solution f(x) satisfying 
f(x+b)=rf(x). 
We can thus say that, for this particular k, the algebraic multiplicity of A* is two, 
but its geometric multiplicity is one! 
If A* corresponds to a closed gap, i.e., A* is a double periodic or antiperiodic 
eigenvalue, then, by Theorem 2, D(A*) = 0 and hence A* is a double zero of F(A; k) 
for a unique (see Theorem 4) k that cquals 0 in the pcriodic case, or equals 7/b in 
thc antiperiodic case. But now, by Theorem B, the geometric multiplicity of A* is 
also two. 
Finally, if A* corresponds to a closed ¢-gap, then again, by Theorem 2, D(A*) = 
0 and hence A* is a double zero of F(A; k) for a unique nonreal k with rem part 
in {0, 7/b}. In this case though (a kind of exception of the exception), as we will 
see later (Theorem 8) there are two linearly independent Floquet solutions with 
multiplier r = ekb; thus, the geometric nmltiplicity of A* is also two. 

4. A CLOSER LOOK AT THE ENDPOINTS OF THE PSEUDOGAPS 
4.1. The operators Lk. Let k  C be given. As we have alrcady seen, the eigen- 
values of the problem 
(46) [a(x)u"(x)]" = Ap(x)u(x), uO)(b) = ru()(O), j = 0, 1,2,3, 
where r = e kb, are the zeros of the function F(A; k) defined in (8). 
Problems such as (46) (almost always related to operators with periodic coeffi- 
cients) have an equivalent formulation (see, e.g., [21]): 
Let Lk be the operator on L2p (0, b) defined by 
(47)  = ()- (/ + ) [()(/, + ) ], 
with periodic boundary conditions. Then v(x) is an eigenfunction of L with cor- 
responding eigenvalue A if and only if u(x) = ekv(x) is an eigenfunction of (46) 
corresponding to the saine eigenvalue A (in other words, u(x) is a Floquet solution 
of (1) with a prescribed multiplier r = eikb). 
The adjoint operator of L k is 
L = L-. 
In particular, Lk is selfadjoint if and only if k  I1. Let Gk(x, y; ) be the Green 
function of Lk and Gk(x,y; ) be the Green function of (46). We have 
Çk(x, y; ) : eik(x-Y)C,k(x, y; ). 
It is well known (see, e.g., [9]) that Gk(x, y; ) can be expressed as an expression 
which is entire in  divided by F(A; k). In fact, Gk(x, y; ) is meromorphic in  and 
its poles are the zeros of F(A; k). If A* is a double zero of F(A; k), then A* can be 
a simple or a double pole of Gk(x, y; ). In the latter case (see [9]), Lk and hence 
(46) may not possess a complete set of (proper) eigenfunctions. More precisely, 
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apart from the eigenfunction, say ¢*(x), that corresponds to A*, there may be a 
generalized eigenflmction b*(x): 

(L - A*) e*(x) = ¢*(x). 

In fact, it is known that if a(x)p(x) -- 1 (thus the problem is essentially of 
second order), then the above situation can actually happen (see [20]). We expect 
this anomaly to arise in the general case too, as long as A* is in an open gap or 
b-gap. These co,uments should be coinpared with Remark 4 above. 

4.2. The endpoints of an open b-gap. The following theorem presents another 
case where au algebraic multiplicity is equal to the corresponding geometric. It can 
be viewed as a partial complcment of Theorem B (in the sense that Theorem B 
is about endpoints of bands, while the theorem below is about endpoints of non- 
touching b-bands). The reinaining case of touching b-bands (equivalently: closed 
b-gaps) is covered later by Theorem 8. 

Theorem 5. Let (,% ) be an open b-gap of (1) (z.e., ( < ) and u --- c or u = . 
Then the Floquet matrix T (u) is similar to the matmx 

( r 1 1 0 0 
0 ri 0 0 
0 0 r 1 1 " 
0 0 0 r -1 

In other words, the equation 

[(x)"()]"= 

bas exactly two linearly independent proper Floquet solutwns, one with multiplier 
r and one with r 1 

Pro@ Let t, = a or . Since (a, ) is a !b-gap and a </, we have that E(t,) = 0, 
but E'(u) ¢ 0. Thus, by (28), 
D(u) ¢ O. 
In the complex A-plane consider the open disk B«(u), i.e., with center u and 
radius e. We choose e > 0 small enough so that 

D(A)¢0 if AeB«(u). 

Let 
ri (u) = e ik(u)b = 
where ik(u) or ik(u - (r/b) is rem and, without loss of generality, strictly positive 
(if not, we consider r5 -I instead of ri). From (4), (9), and (16), 
1 A(A) et: V/ÊTA) j = 1 or 2, 
r(A) + r(A = -- 2 ' 
where vf denotes the principal branch of the square root function. If 
then E(A) -- E(A)(A - u), where E(A) # 0 in B«(u). Hence the above formula 
can be written as 
1 A(A) 
r,(A) 2 2 
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and 
1 A(A) V/Ê- /- _ oe. 
(48) T() + - 
T(),) 2 2 
Assuming Il(,)l, Ie(')l > 1, the above equations give r(A) = e ik()b and r2(A) = 
e k() uniquely. Of course, kl() 
If « is sufficiently small, Theorem 1 implies that F(; k(v)) h exactlv one zero 
in B«(v), namely  = v. In fact, there is a neighborhood N of k(v) such that, 
if k e N, then F(; k) h exactly one zero (counting nmltiplicities)  = (k) 
in B«(v). Furthermore, if k  N, k # k(v), then (1) bas four proper Floquet 
solutions corresponding to this  = A(k), with corresponding multipliers r(A) = 
¢ikb, rl()-I = ¢--ikb, T2(), and re(A) -1, where r2() is given by (48). Therefore, 
for k  N, k # k(), we must have (since one of these four Floquet solutions 
corresponds to an eigenfunction of Lk) 
G(x, x; )dx d = 1. 
where,  in the previous subsection, Gk(x, y; ) is the Green function of La. Letting 
k  k(oe) we obtain 
 G()(x, x; )dx d = 1, 
2i 
which says that there is only one (proper) Floquet solution corresponding to ri (oe) = 
r2(oe) = e ik()b (if there were two Floquet solutions, the value of the integral in 
(48) would have been 2). Considering the adjoint case, which h an equivalent 
behavior (see [9], Ch. 12, Sec. 5), we can conclude that there is, also, only one 
Floquet solution corresponding to ri 
Remark 5. If E (oe) = 0, oe  0 (remember that oe  ), then, since Lk and its adjoint 
L have the saine number of proper and generalized eigenfunctions corresponding 
to oe (see [9], Ch. 12, Sec. 5), it follows that T (oe) is similar to one of the following 
matrices: 
(0 0)(100 0 
0 r 0 0 0 r 0 0 
0 0 r  1 ' 0 0 r  0 " 
0 0 0 r 1 0 0 0 r 1 
Theorem 5 covers the ce where 
is a double zero of E (), i.e., the corresponding -gap is closed, is deeper, since in 
this ce (49) becomes 
2i ( G(,i(z,z; )dz d = 2. 
Now, by Theorem 1, u is a double zero of F(; k(u)). Consequently, the above 
equation says that either L(,) h two proper linearly independent eigenfunctions 
and hence T () is diagonalizable, or (, has one proper and one generalized 
eigenfunction and hence T (u) is not diagonalizable. Later, in Theorem 8, we will 
see that the latter can never happen, i.e., T () is always diagonalizable. 
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5. A MULTIPOINT EIGENVALUE PROBLEM 
In the Hill case, the Dirichlet spcctrum {#n}n__l (i.e., the eigenvalues corre- 
sponding to the boundary conditions u(0) = u(b) = 0) plays an important role in 
the general spectral theory, especially in the formulation and solution of the inverse 
spectral problem. We propose the following multipoint problem as an mmlogue of 
Hill's Dirichlet problcln for the Euler-Bernoulli case: 
(50) [a(Æ)u"(x)]" = Ap(x)u(x), u(O) = u(b) = u(2b) = u(3b) = O. 
An eigenvalue of (50) is any valuc of A for which (50) has a nontrivial solution. We 
call such a solution an eigenfunction of (50). 
Physically thc problem (50) describes the vibration of a (periodic) beam fixed 
at four points. 
Let uj(x;), j = 1,2,3,4, be the flmdamental solutions of (1). Since every 
solution of (1) is a linear combination of the flmdamental solutions, it follows that 
A is an eigenvalue of (50) (that is, A is such that (50) has a nontrivial solution) if 

and only if ,k is a zero of the entirc flmction 

tl(0; A) t2(0; A) U3(0; A) U4(0; A) 
U« (b; A) u2(b; A) t3(b; A) t4 (b; A) 
ul(2b;) u2(2b;) u3(2b;) u4(2b;A) 
ul(3b;A) u2(3b;A) u3(3b;A) u4(3b;A) 

But ul(0; A) = 1 and u2(0; A) = u3(0; A) = u4(0; A) = 0; therefore, 
u2(b; A) u3(b; A) u4(b; ) 
(51) H(A)= u2(2b;,k) u3(2b;&) u4(2b;&) 
u2(3b;,k) u3(3b;&) u4(3b;&) 
We can, thus, say that the spectrmn of (50) is the set of zeros of H(&). In particular, 
H(O) = b  dx . 
a(x) ' 
thus 0 is not an eigenvalue of (50). In the unperturbed ce we have 
(52) 
»:. (''b) [ch(l'b) <,b)]: 
Notice that by (31), 
0() - D0 () 
16b 2 
We continue with some properties of (50) and its accompanying function H(A) 
But first we need some lemm. 

Lemma 3. (a) If there is a nontrivial function u(x) such that 
[a(x)u"(x)]"= ,kp(x)u(x), bi < x < b2, 
u(b) = u'(bi) = u(b2) = u'(b2) = 0, 
then , is real and strictly positive. 
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(b) Likewise, if there is a nontrivial function u(x) such that 
[«(x)u"(x)]"= Ap(x)u(x), x e (b, oc) (or x e (-oc, b)), 
u(b) = u'(b) = O, u e L2(b, oc) (resp. u e L2(-oc, b)), 
then A is real and strictly positive. 
Proof. (a) Multiplying the equation by u(x) and integrating yields 
b [a(x)u"(x)]'u(x)d« = A p(x)u(x)u(x)dx. 
1 1 
Next, by applying integration by parts (twice) in the left-hand side and using the 
boundary conditions, we obtain 
1 Jbl 
The assumption that u(x) is not trivial, together with the boundary conditions 
u(b) - u'(b) = 0, imply that u(x) is hot a liuear fimction. Thus both integrals in 
the above formula are strictly positive (rcmcmber a(x), p(x) > 0, for all x): hence 
A>0. 
(b) The proof of this part is very similar to the proof of part (a). Assume that u E 
L 2 (b, oe). Since u(x) is a linear combination of Floquet solutions (possibly including 
generalized ones), it follows that u(x) and its derivatives decay exponentially, as 
x -- oe; thus, we can apply again integration by parts and get 
 a(x)u"(x)'--dx =  (x)(x)(x)dx; 
hence, again A > 0. [] 
The following lemma is contained in [31]. We include it here for the sake of 
completeness. 
Lemma 4. Let A > O. If the Floquet multipliers are indexed so that Irll >_ If21 >_ 
]r31 _ [r4], then (r3 = r 1 and) 
--1 
(53) ri > If21 > If31 > r4 = ri 
Furthermore, the Floquet solutions f (x) and f4(x) corresponding fo r and r4 never 
vanish. 
Proof. If A > 0, u (x; A), the first fundamental solution of (1), and u (-x; A) are 
increasing when x _> 0. They actually grow exponentially. If A is in the spectrum 
of (1), Ir21 = Ir31--- 1 and thus (53) and the statement about Il(x) and f4(x) must 
be true (remember f(x) = eWXpi(x), where p(x) is b-periodic and r - ewb); 
otherwise, there would not be any exponentially growing solutions. Similarly, if A 
is not in the spectrum, r2 and r3 = r - are real. If we take the period of (1) to be 
2b, then the Floquet multipliers become 
But the above inequalities become equalities only if A is a zero of E(A). Since A > 0 
and the zeros of E(A) are nonpositive, we must have 
Irai > If21 > Ir3I > If41- 



3748 VASSILIS G. PAPANICOLAOU 

Now 
l I (x) = Clfl(x) ÷ c2f2(x) ÷ c3f3(x ) ÷ c4f4(:T). 
Hcnce the exponential growth of 1 (X) and ut (-x) implies that ri and r4 are 
positive and fl(x) and f4(x) do not change sign. [] 

The next theorem should be compared with the property of the Hill operator 
stating that the Dirichlet eigenvalues are simple and their corresponding eigenfunc- 
tions are Floquet solutions [39]. The case left open (namely when p is also a simple 
periodic or antiperiodic eigenvahm) is covered later by Theoreln 7. 

Theorem 6. Let p be an eigenvabte of (50). If I'(p) denotes the corresponding 
eigenspace, namely the vector space of all eigefunctions of (50) associated fo p, 
then dimV(p) = 1 or 2. Furthermore, t'(p) always contains a proper Floquet 
solutiom if diln V(p) = 2, then V(p) always contains two linearly independent 
proper Floquet solutions, except possibly in the case where p is also a simple periodic 
or antiperiodic eigenvalue of (1) (zn fact, we will see later, in Theorem 7. that this 
exception can never happen). 

Pro@ Let H(p) = 0. Assulne that dilnV(p) = 3 (clearly dimI'(p) < 4 ). Then we 
have three linearly independent eigenflmctions ¢ (x), ¢(x), and ¢3(x) correspond- 
ing to p. Let 

¢(X) -- C1¢1(3C) ÷ C2¢2(X) ÷ C3¢3(X)- 
= ¢(b) = 0. Also, we can choose ci, c2, and c3 (hot ail zero) 
= b'(b) = 0. Hence, Lelnma 3 implies that # > 0. But then, 

We have ¢(0) 
so that ¢'(0) 
Lemma 4 implies that the associated (to #) Floquet solutions ft (x) and f4 (x) 
never vanish. Furthermore, the space spanned by ¢(x), ¢2(x), and ¢3(x) and the 
space spanned by f (x) and f4 (x) must have a nontrivial intersection. Thus there 
is an eigenfunction of (50) of the form 

"Ylfl (x) ÷ "Y4f4 (x). 

But this implies easily that 

f, (0; p) fa (0; #) = O, 

a contradiction. Thus dimV(#) < 3. 
We now prove the rest of the theorem (see Theorem B and formula (16) for the 
definition of F +, F-, and E that appear below). 
Case F+(#)F-(p)E(p) 7 O. Then, for A = p, (1) possesses four distinct Floquet 
multipliers ri = ri(p), j = 1, 2, 3, 4; therefore, it has four linearly independent 
(proper) Floquet solutions fj(x) = fj (x; p), j = 1, 2, 3, 4, with 

fj(x+b)=rjfj(x). 

If, in addition, # is in the spectrum of (50), then 

f,(o) f(o) f3(o) 
ri (b) f2 (b) f3 (b) 
ri (2b) f2(2b) f3(2b) 
f (3b) f2(3b) f3(3b) 

f,(o) 
fa(b) 
f4 (2b) -- 0; 
fa (3b) 
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namely, 

fl (0; p) f2 (0; p) f3 (0; p) fa (0; p) 

or 

fl (0; #) f2 (0; #) f3 (0; #) fa (0; #) 

But the rj's are distinct; thus, 

1 1 1 
ri r2 r3 

1 

1-[ (r - ri) = 0. 
l_<j</_<4 

(54) fl (0; p) f2 (0; p) f3 (0: p) fa (0; p) -- 0. 
This means that some Floquet solution, say f (x; p), is an eigenfunction of (50). 
Next, let ¢(x) be another eigenfunction of (50) corresponding to p. That is, ¢(x) 
and fl (x) are linearly independent. It follows that there is a constant ci such that 
p(x) = ¢(x) - clf (x) is an eigenfun«tion of (50) and 
(x) = c2f2(x) + c313(«) + «14(x). 
But this implies easily that 
f2 (0; #) f3 (0; #) fa (0; #) = 0, 
which, again, means that some Floquet solution fj, j = 2, 3, 4, say f2 (x; #), is an 
eigenfunction of (50). 
Case E(#) = 0. That means that # < 0 and ri ---- r2 = r - = r -1. If we 
have coexistence of two Floquet solutions fl (x) and f2(x) with nmltiplier r, then 
(see Remark 5) we also have coexistence of two Floquet solutions, f3(x) and fa(x), 
with multiplier r -1. We can then find constauts ci, c9, c3, and c4, such that 
off(x) + cgf2(x) and c3f3(x)+ c4fa(x) are in V(#). If we do hot have coexistence, 
then (see Remark 5) we bave two proper Floquet solutions f(x) and f3(x) with 
corresponding multipliers ri and r -1, and two generalized Floquet solutions gl(X) 

and g3(x) satisfying 
(55) 
g (x + b) = rlg (x) + dfl (x) , 
Since H(#) = 0, we must have 
fl(0) 
f (b) 
f (25) 
f (35) 

which implies 

g3 (X q- b) -1 
= ri 93 (x) Te d3f3 (x), dld3 # O. 

g (o) f3 (o) 3 (o) 
gl (b) f3 (b) g3 (b) 
gl (2b) f3 (2b) g3 (2b) 
gl (36) f3 (36) g3 (36) 

=0, 

fl (0; #)9. f3 (0; #)2 _-- 0. 

The last equality says that one Floquet solution, say fl (x), is also an eigenfunction 
of (50). If ¢(x) is another eigenfunction of (50) corresponding to #, then there is 
an eigenfunction of the form 
((X) = «igl(X) q'- c3f3(x) - «4g3(x). 
This implies that g(x) E 1/'(#) or f3(x) E 
From (50) and the fact that f (x) bas the form 
(56) fl(x)=epl(x) with rl=e  and pl(X+b)=pl(x), 
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it follows that gi(x) has the form 
(57) 
gl(x) : [p2(x) " /pl(,7)æ]e wlag 

with  = __d and p2(x+ b) = p2(x). 
rlb 

Let us assume gl(x) E V(#). 
pi(O) = p2(O) = O: therefore, 

Since we also have g(x)  V(#), it follows that 

Next notice that 

fl (,,b) = gl (T/b) = O, 

for ail n  Z, 

for ail nZ. 

(59) [a(x)v"(x)]"= pp(x)v(x) 

and (for all n  Z) 
(60) v0,b) = 0, v'(0) = 0. 

By (59), 
nb  fonb 
o [a(x)¢'(x)]"(x)dx = p(x)()()d. 
Then, integration by parts and (60) yield (recall that a(nb) = a(0)) 

(61) 

-a(O)v"(b)v'(b + fo 'b 

If we set 

fo nb 
a(x)v"(x)v"(x)dx = p p(x)v(x)v(x)dx. 

{-1, if Ir[ > 1, 
e = e(ri)= 1, if Ixl < 1, 
then (56) and (57) imply that f(z), 9(z), and their derivatives decay exponen- 
tially, as z - eoe, and by (58) the saine is true for v(z). Thus (61) implies that 
in particular, # > 0, a contradiction. Therefore, g (x)  V(#) and we are leff with 
the only alternative, namely that f3(x)  V(#). 
Case F+(#)F-(#) = 0. That means that # > 0 is a periodic or antiperiodic 
eigenvalue. If we have coexistence of two periodic or antiperiodic Floquet solutions, 
then a linear combination of these can produce an eigenfimction of (50), and there 
is no other eigenfimction, i.e., dimV(#) = 1 (since, by Lemma 4, the other two 
Floquet solutions never vanish). If there is only one periodic (or antiperiodic) 
Floquet solution, say f2 (x), then there is a generalized Floquet solution g2 (x), 
satisfying 
g2 (x + b) = «g2 (x) + c2f2 (x), c2 ¢ O, 

so that 

' = ,p,(0) ¢ o and ,(,,b) 1 [@(0) + pi(O)] 
where fil (0) # 0 follows from Lcmma 3. Introduce 
(58) v(x) = p(O)fl (x) -/)'1 (O)gl (X), 
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where « = 1, if F+(p) = O, and e = -1, if F-(p) = O. 

implies that 

fl (0) 
ri (b) 
ri (2b) 
ri (3b) 
which implies easily that 

A(o) 2(0) A(o) 
f2(b) g2(b) f4(b) 
f2 (2b) 92 (2b) f4 (2b) 
f2 (3b) 92 (3b) f4(3b) 

In this case, H(p) = 0 

=0, 

hence there is a Floquet sohition 
(this is, however, iinpossible, as we will see later in Theorem 7), we cannot, for the 
moment, exclude the possibility that g2 () 
Remark 6. One part of Theorem 6 states that the geometric multiplicity m,p) of 
any eigenvalue p of (50) cannot exceed two. We can define the Mgebraic multiplicity 
m(p) of p to be its multiplicity as a zero of H(A). )om the above proof it follows 
esily that m(p) OE mg(p.). Theorems 7 and 8 below establish the equality of the 
two multiplicities. 
The lemma that follows is needed for the proof of Theorem 7 below. 
Lemma 5. There are no zeros of ri(A) in the interior of the bands or in the interior 
of the ¢-bands. 
Proof. Assume that p is in the interior of a hand. Then F+(p)F-(p)E(p)  0: 
hence (54) must hold. By Lemma 4 we have that fl (x; p) and f4(x: p) never vanish, 
where r3(p)= r(p)= r2(p) ¢ . Also, f3(x;p)= f(x;p). Therefore, 
(62) f (0; ) = f (0 ,) = 0. 
We can, thus, write a linear combination 
f (x) = d2f(x; p) + d3f3(x; p) 
such that f(0) = ff(O) = 0. This means that f(x) can be written  a linear 
combination of the fundamental solutions u(x: p) and u4(x; p), lmmely 
(63) f(x) : 33(X; p) + 44(X; p). 
But (see [31]) u3(x; p)  oe, as x  oe, while u4(x; p)  oe,  x  oe. Since 
f(x) is bounded we can conclude that (63) is impossible. Thus (62) is impossible 
and H(p)  O. 
Assmne now that p is in the interior of a ¢-band (hence p < 0). Then there is 
a k  (0, /b) such that 
rl (p)- 1 1 
-- T2 ()-  --[rl ()[E ikb. 
,() () 
If H(g) = 0, then (54) implies that f (0 g), for some j. Let us assume that 
We have 
f (x;p)=eeikp(x) and f(x;p) = f (x;p)=ee-kp(x), 
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where c E R \ {0} and Pi (x ÷ b) = pi (x). Thus 
I2(0; =0 
and fl(';#), f2(';#) e L2(0, Oe) (if c < 0) or fl(';#), f2(';#) e L2(-Oe,0) (if 
c > 0). We can, therefore, write a nontrivial limar combination 

u(x) = cf (x; p) ÷ Clfl (x; #) 
that satisfies all assumptions of part (b) of Lemnm 3 (with b = 0). Hence # > 0, a 
contradiction. Therefore, H(#) 

We are now ready to prove our main theorem regarding the spectrum of (51), 
i.e., the zeros of H(A). The statement of the theorem resembles the one of Theorem 
2. 

Theorem 7. Ail zeros of H(A), of (51), are real and they are located as follows: 
(a) H(A) bas exactly one (simple) zero in the closure of each gap of the spectrum 
S(a, p) (with the understanding that, if the gap is close& i.e., collapses fo a double 
periodic or antiperiodic eigenvalue, say A*. then the simple zero of H(A) is *); (b) 
H(A) bas exactly two zeros (counting multiplicities) in the closure of each ¢-gap of 
the pseudospectrum. In case (b), if the ¢-gap is close& i.e., collapses fo a point 
v* where v* ' for some n = 1, 2, 3, (see the 
 U2n--i U2n, OT U*  U ! 
: - : U2n--i 2n "'" 
statement of Theorem D), then v* is a double zero of H(A). There are no other 
zeros of H(A). 

Pro@ As in the case of Theorem 2 (see also the appendix), we only need to prove 
the theorem for the case of smooth a(x) and p(x). Hence, from now on we assume 
a, p e C(l). 
By (52) the theorem is valid in the unperturbed case a(x) =- p(x) -= 1. For general 
a(x) and p(x) let us consider the deformation (19) and the solutions bj(x: A; t), 

j = 1, 2, 3, 4, satisfying (22). We set 

(64) #(A: t) = 

¢1(0;t) ¢(o;t) ¢3(0;t) 
¢(b; t) ¢(b; t) ¢(b; t) ¢(b; t) 
çbl(2b;t) çb2(2b:t) çb3(2b;t) çb4(2b;t) 
¢(3b;t) ¢(3b;t) ¢(3b;t) ¢(3b;t) 

where the dependence of Ci in A has been suppressed for typographical convenience. 
For small lAI's, the Ci's can be chosen so that H(0; t) stays away from 0, uniformly 
in t. Notice that /Ï(A; t) and H(A; t) have the saine zeros. The multiplicities of 
their zeros also agree (this can be easily checked when t = 0; then, since t is moving 
continuously, the multiplicities of two corresponding (i.e., equal) zeros, one of H 
and one of , cannot suddenly become different). 
In order to use the technique described in Subsection 3.1, we first need to estimate 
the large-magnitude zeros of H(A; t). First, consider {#,(0)},ez, the set of zeros 
of/(A;0), counting nmltiplicities. As we have seen, (A;0) and Do(A) have the 
saine zeros; thus, (35) gives 

4 
#1-2/(0) --- p-2/(0) -- --4 () , P2/-1 (0) --- P2/(0) - , 1 = 1,2,3 ..... 
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If p is a large-magnitude zero of (A; t), then (22) and (64) imply that 

(uniformly in t), where ci,et E {i,-1,-i, 1}, ej  et. This, in turn, implies that 
there is a K > 0 (independent of t) such that 
ip/,_ p(0)i/, I < u for some n E Z, 
-i1/,, 
from which it follows that 
Iv- (0)1 </-2, foÆ some  e 2. 
In other words, if H(p; t) = 0 and IPl is sufficiently large, then there is an integer 
n such that p is within distance Kn 2 from p(0). On the other hand, as we have 
already seen in (42), there is a constant C > 0 such that 
#,+2(0) - #,(0) > C Inl 3 , for ail n e Z. 
Therefore, as in the proof of Theorem 2, no new zeros of H(A; t) tan corne from 
infinity, as we more t. 
Thus, by the above discussion and Lemma 5, thc only way in which the theorem 
tan be violated is if some zeros of H(A; t) become nonreal. As we start moving t, 
the zeros in the gaps (including closed gaps) are all simple. Thus, the zeros that 
ean first leave the real axis (in pairs of complex conjugates, of course) are the zeros 
in the ¢-gaps. 
Let (,/3) be a ¢-gap (, /3 vary continuously with t). For A in [, fil, let ri, 
r2, r3 = r -1, and rl = r -1, with Irll _> if21 > 1, be the corresponding Floquet 
multipliers of 
la(x; t)"(x)]" = p(x; t)(x). 
Since (a,/3) is a ¢-gap, there is a 5 > 0, independent of t, such that 
Irai > If21 E l + ,$. 
Nowlet D  C be adomain (dependingon t) such that [,/3]  D. IfD is 
sufficiently small, then, as functions of A, fj(x;A;t) and, in particular, fj(O;A;t), 
j = 1, 2, 3, 4, are analytic in D, with the only singularities being the branch points 
a,/3 (if a ¢ ff). As A moves around one of these branch points, ri (x; A) becomes 
f2(x;A) and fi(x; A)becomes f3(x;A). 
Initially (i.e., when t = 0), p =  =/3 is a double zero of H(A; 0) and there are 
two (linearly independent) Floquet solutions f(x; p) and f4(x; p), with multipliers 
ri and ri respectively, satisfying fl(0;p) = fii(0;p) = 0 (fl(x;p) and f4(x;p) are 
eigenfunctions of (50) corresponding to the eigenvalue p). As we move t, f (0; A) 
(or f2(0; A), if we encounter a branch point) will continue to have a zero in [a,/3], 
and so will f4(0; A) (or f3(0; A), if we encounter a branch point). In order for a 
zero of H(A; t) to escape from the rem axis, it first has to become double. So let us 
assume that, for t = to, # = #(to) G l satisfies fj(O;#(to)) = ft(O;#(to)) = O, j  l, 
i.e., #(to) is a double zero of H(A; to). Notice that f(x) and ft(x) must belong, 
one to L2(-Oe,0) and one to L2(0, Oe). This follows by continuity, but, also, by 
Lemma 3 (if both f(x) and ft(x) are, say, in L2(0, oe), then we can construct 
a nontrivial solution v(x) = cf(x) + ctft(x), such that v(0) = v'(0) = 0 and 
v  L2(0, oe), which implies that #(to) > 0, a contradiction). Thus, without loss 
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of generalitv, we can ta.ke j = 1 and I = 4. Assmne that, as t gets larger than to, 
then imlnediately p(to) splits into two nonreal zeros p(t) and it(t) of H(A; t). The 
corresponding Floquet solutions are fl(x: It(t)) (or f2(x:p(t)), if p(to) is a branch 
point) and f4(x; p(t)) (or f3(x', p(t)), if p(to) is a branch point). They also have to 
be conplex conjugates. But this is a contradiction, since, on the one hand, f(x), 
f2(x) are in L2(-oe, 0) and fs(x), f4(x) are in L2(0, oe, ); but, on the other hand, a 
[unction and its COnlplcx conjugate are in the saine L2-space. Therefore, the zeros 
of H(A: t) can nevcr leave the real axis and the theorem is proved. [] 

Remark 7. Since, b.v the abovc theorcln, all thc positive zeros of H(A) are simple, 
the ceuse in Thcorcm 6 that was left unanswered, nalnely when p is also a simple 
periodic or antipcriodic cigenvalue, is now dccided: We always have dinfl'(p) = 1. 

Reznark 8. If the coeffiçients of (1) are even fimctions, namely if 

.(-«) = () ,,d p(-) = (), 

then for every solution u(x) of (1) we llave that t,(x) = u(-x) is also a solution. In 
particular, if f(x) = ep(x) is a Floquet solution, so is f(-x) = e-p(-x). It 

follows that, in this case, the 
of (50), must also be periodic 
of H(A) are all double, since, 
eigenflmctions of (50), f(x) = 
and the other in L2(0. 

positive zeros of H(A), i.e., the positive eigenvalues 
or antiperiodic eigenvalues, while the negative zeros 
if p < 0 is such tha.t H(p.) = 0, then there are two 
e'p(x) and f(-x) = e-'p(-x) (one in L2(-oc,0) 

Remark 9. For { G ]R, consider the one-paralneter falnily of shifted or translated 
functions a{(x) and p{(x) of a(x) and p(x), namelv 

(65) 

a(x)=a(x+) and p(x)=p(+), 

and let T(A;{) be the corresponding Floquet matrices. Then T(A: {) is sinfilar to 
T()Q, for all {. In particular, this one-parameter family is isospectral and "iso- 
pseudospectral". On the other hand, the spectrmn {Pn({)}-ez of (50) evolves with 
{ (thus we have an isospectral and iso-pseudospectral flow). It will be interesting 
to understand the evolution of p,({)'s with ¢, since this might provide the solution 
to the inverse spectral problem. More generally, we would like to do the analysis of 
(1) froin the point of view of [12]. Che relevant observation here is that, if (a,/) 
is a g,-gap and p is in [a,], then there is a { such that p is in the spectrum of 
the nmltipoint problem for a{(x) and p{(x). This is because, if ) G la,ff], then (1) 
always has a Floquet solution, say fj(x; )), that vanishes for some x = x0. Taking 
{ = x0 does the job. 
Vfe believe that the saine is truc for the gaps of the spectrum; namely, that for 
any A in a gap, there is a  so that A is in the spectrum of the multipoint problem 
for some a¢(x) and p{(x). 

The last theorem of this article is an application of Theorems 6 and 7. It com- 
pletes Theorem 5, so that the two theorems together form the analogue of Theorem 
B for the pseudospectrum. 

Theorem 8. Let u* = u,_ = u, or u* = u2.-i = u2; namely, u* is a collapsed 
(i.e., closed) ¢-9ap of (1) (equivalently. u* is a double zero of E(A)--sec Theorem 
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D). Then the Floquet matrix T (p*) is similar to the 

r 0 0 0 
0 ri 0 0 
0 0 r  0 
0 0 0 r  

In other words, for the equation 

[a(x)u"(x)]" 

=p*p(x)(x), 

EQUATION 
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we have coexistence of four Floquet solutzons, two with multiplier ri and two with 
multiplier r 1 
Proof. Consider the shifts a¢(x) and pe(x) of a(x) and p(x) of (65). As we observed 
in Remark 9, the equation 
(66) [a¢(x)u" (x)]" = Ap¢(x)u(x) 
has the same spectrum and pseudospectrum as (1), for ail   . In particular, 
p* is a closed ¢-gap of (66), for all   I. The Floquet multipliers rj = ri(p* ), 
j = 1, 2, 3, 4, are the saine for all    and we bave 
r = r = r  = rj . 
Next we considcr the multipoint eigenvalue problem 
(67) [a¢(x)u"(x)]" = p¢(x)u(x), u(O) = u(b) = u(2b) = v(3b) = 0. 
From the above discussion and Theorem ï it follows that oe* is a double eigenvalue 
of (67), for all   . Thus, by Theorem 6, for any   , there are two Floquet 
solutions, f,¢(x: p*) and f4,¢(x; p*) of (66), with corresponding multipliers r and 
r4 = r - such that 
(68) f,¢(0; p*) = fa.¢(0: p*) = 0. 
Each Floquet solution of (66) is a -shift of a Floquet solution fj(x; A) of (1): 
f,(x;p*) = f(x + ;p*) and 
Hence, if we did hot have coexistence, then (68) would imply 
fl(,; v*)= f(: oe*)= O, for a.ll 
which is impossible since the Floquet solutions considered are nontrivial. Thus we 
have coexistence of two (linearly independent) Floquet solutions with multiplier r, 
and two Floquet solutions with multiplier r -1 [] 
Remark 10. The above theorem, together with Theorems 6 and 7, imply, in par- 
ticular, that mg(p) = m(#), for any eigenvalue # of (50) (see Remark 6). 
In relation to Theorem 8 we notice tha.t, if for some 
is similar to the diagonal matrix diag(r, r, r -. r -1 ), then T (p*) has a very special 
structure. Its sixteen entries must sa.tisf3  various relations. 
Finally, we want to mention a coujecture and three open questions. 
Conjecture. If all nonzero zeros of E(A) are double (see Theorem D), then 
p(x)a(x) =- 1. Equivalently: if ail the ¢-gaps are closed, then the beam operator zs 
a perfect square of a Hill-type operator. 
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Open Question 1. An interesting question is: What can be said about a(x) 
and p(x) if we know that the spectrum S(a,p) of (1) has no gaps; namely, if 
s(a,) = [0, )? 
Open Question 2. In the Hill case there is a sinple correspondence between 
periodic inverse spectral data and inverse spectral data of the separated boundary 
value problem on the interval (0, b), where b is the period of the potential (see [14]). 
Find the analogous correspondence for the beam operator. This will be a major 
step in the solution of the inverse periodic spectral problem for the beam. 
Open Question 3. Extend the results presented in this paper to n-th order op- 
erators. 

APPENDIX 
We present here a proposition which was used in the proofs of some of our 
theorems. 
Proposition. For an, Pn  C¢[0, b], an(x), pn(X) >_ m > 0, n = 1.2,3 .... , 
consider the initial value problems 
(69) [an(x)u(x)]"--- Apn(x)un(x), 0 < x < b, 
(70) un(O;A)=, u(O;) , " '"'' 
= un(0;A)=% unw;A)=& 
where pmmes denote derivatives with respect fo x and A  C zs a parameter If 

(71) 
where 

C 2 
an(x) ---, a(x) and pn(x) c p(x), 
then 
n(b; A) - (b: ), 
uniformly on compact subsets of C, where 
[a(x)u"()]" = ()(), 
u(0;)=, '(0;)=, "(0;)=, 
Proof. We write (69)  a first-order system 
dy 
- A (x; A) y, 
dx 

0<x<b, 

u'(;) o 
y (x;A) = an(x)u(x;A) ' An (x;) = 0 
The initial conditions (70) become 

yn (o; A) = 

Consider also the problem 
(72) 
(73) 

an(0) " 
a(O) + an(O) 

--- = A (z; ) y, 0 < x < b, 
dx 
y(0;) = n (0;), 

100) 
0 an(x) - 0 
0 0 1 " 
0 0 0 
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where 

A(x;A) = limAn (x;A)= 
n 

/ 0 1 0 0 / 
0 0 a(x) -1 0 
0 0 0 1 " 
,p(x) o o o 

Let us assume that lAI _< M, where M is a fixed positive number. If we set 

n (x;) = n (x;)-  (x; A), 

then by (71), (72), and (73), 
dwn 
dx -- Anyn - Ay = A,wn + (An - A) y, 
wh (o; A) = o. 

0<x<b, 

It follows that 

Thus 

An (¢; A)a'n (¢; A) de +/o x 

[A, (¢; A) - A (¢; )] y (¢; ) 

(74) 
where 

Notice that 

0 x 
Iwn (x;,X)l _en ÷ IIAn(;X)lllwn(;X)ld, 

./ôO b 
en= 

(75) 

lim en = 0. 
n 

Applying the Gronwall inequality (see [9], Ch. 1, Prob. l) to (74) we get 

0 b 
an IIAn (;)11 is bouned for  e [0, bi ad I1 -< M (unformly in). Terefore, 
by (75) we have that 

Wn (b;A) = Yn (b;A) - y (b; A)  0 

uniformly in A, as long as lAI _/I. Since/I is arbitrary, the proof is complete. [] 
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SINGULARITIES OF THE HYPERGEOMETRIC SYSTEM 
ASSOCIATED WITH A MONOMIAL CURVE 

FRANCISCO JESIS CASTRO-JIMÉNEZ AND NOBUKI TAKAYAMA 

ABSTRACT. We compute, using/9-module restrictions, the slopes of the irreg- 
ular hypergeometric system associated with a monomial curve. We also study 
rational solutions and reducibility of such systems. 

1. INTRODUCTION 
Let An = C(xl,... ,xn,(91,...,(9,,) be the Weyl algebra of order n over the 
complex numbers C and let C[(9] = C[(91,...,(0n] be the subring of An of linear 
differential operators with constant coefficients. 
Let A = (aij) be an integer (d × n)-matrix of rank d. We denote by IA C C[(9] 
the toric ideal associated to A: i.e., IA is the ideal generated by the set 
{ Ou - Or I u, v E N n, Au T = Av T } 
where ()T means "transpose". 
We denote by 0 the vector (0,...,On) T with Oi = xi0i. For a given 
(f,..., rg) T  C d we consider the column vector (in Ado) AO - f and we denote 
by (AO - f> the left idem of An generated by the entries of A0 - 
Following Gel'fand, Zelevinskiï and Kapranov [6], we denote by HA(ff) the left 
ideal of An generated by IA t2 (AO - ff>. It is called the GKZ-hypergeometric system 
associated to the pair (A, f). The quotient ?-lA(f) = An/HA(f) is a holonomic 
module (see e.g. [1 71). 
If the toric ideal IA is homogeneous, i.e., if the Q-row span of A contains 
(1,..., 1), it is known ([8]; see also [17]) that 7-fA(f) is regular holonomic and the 
book [17] is devoted to an algorithmic study of such systems. Especially, the book 
gives an algorithmic method to construct series solutions around singular points of 
the system. 
In this article, we start a study of singularities of GKZ-hypergeometric systems 
for non-homogeneous toric ideals IA by treating the "first" case when d 
(al,a2,...,an)  Z , al : 1. We evaluate the geometric slopes of 7-(A(f) by 
successive restrictions of the number of variables. The slopes characterize Gevrey 
class solutions around a singular locus. Let us remark on an analytic meaning 
ofslopes. Let X = C n and Y = {x I xn = 0} C X. We denote by O the 
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forlnal cornpletion of O\- along Y (i.e., series t.hat are forrnal iii Xn and convergent 
in Zl,...,z-l). To each real number s  [1,+oo) we denote by Ovly(S) the 
subsheaf of O.. of Gevrey functions of order s (along Y). The sheaf Oxlr(1) 
is the restriction Oxl" a.nd, by defilfition, we write Oxly(+Oo) = O.-:ïi.. For any 
holononfic 79x-module AI, Z. Mebkhout [14] associates the sheaf Irry (s) (Al) as 
the solution sheaf RHornz)(M, Oxw(s)/Oxw). One fundarnental result in the 
irregularity of 79-1nodules is the fact that Irr-(s)(-Al) is a perverse sheaf, for any s 
(see [14]). These sheaves define a filtratiou of the irregularity of AI along Y, i.e., 
Irr-(M) := Irr-(+ec)(-Al). The main result of [11] is that 1/(1 - s) is a slope 
of _AI with respect to t if and only if s is a gap of the graduation defined by the 
filtration on the irregularity. In other words, 1/(1 - s) is a slope if and only if 
Irrr(s)(M)/Irrr(< s)(-Al)¢ 0. 
Out evahlatioli of the slopes is done as follows: (1) \Ve translate Laurent and 
iMebkhout's theoreln [12] on restrictions and slopes of 79-modules into an algorithm 
to evahmte the slopes by utilizing the results of [2] and [15]. (2) Apply out general 
algorithln to the hypergeometric systen, associated to A = (1.a2, .... a,). This 
systern has iuany nice properties and out algorithrn outputs the slopes without 
COllput.ation on COlnputers. 
In the last section we study ratiolml solutions and reducibility of out systems. 

2. IIICRO-CHAIRACTEIRISTIC VARIETIES 

In this section, following Laurent [10], we describe inicro-characteristic varieties 
for a given 79-module. \Ve will state a result of Laurent and Mebkhout [12, Corol- 
laire 2.2.9] (see also [14, p. 125] and [11, p. 42]), allowing to reduce our general 
problern of evaluating the slopes to fewer variables. 
In this section X = C  and 79x = 79 is the sheaf of linear differential operators 
with holornorphic function coefficients. Let -AI be a coherent 79-module. 1Recall 
that the characterist.ic variety of M (denoted by Ch(-AI)) is an analytic subvariety 
of the cotangent bundle T*X. 
Suppose that Y C X is a srnooth hypersurface. \Ve say that }" is non-characterzs- 
tic for _Al if TX  Ch(_AI) C T.-X. Here T-X is the conorrnal bundle to " in X 
and T.X is the zero section of T*X. 
Now, following Laurent [9], [10], we shall define the notion of non-micro-charac- 
teristic varietl for _AI. To sirnplif.v the presentation we will assume that (z, x2 .... , 
xn) are local coordinates in X and that }" is defined by xn = 0. We denote by 
(Xl,...,x,,l,...,) local coordinates in T*X. Sornetirnes it will be useful to 
write x = y,... ,Xn_ 1 = Yn-, Xn = t,  = rl .... ,n- = rln- and son = T. 
Let us denote by A the conorrnal bundle of Y in T*X (i.e., A = T¢.X). So, in 
local coordinates, A = { (y, t, r/, r)  T*X If - r/= 0}. Here, r/= (r/l,..., r/-l) and 
y = (yl,-.. ,y,-l). We denote by ('y, r, y*, r*) local coordinates on the cotangent 
bundle T*A. 
We denote by 1.(79) (or simply by V) the Malgrange-Kashiwara filtration asso- 
ciated to Y on 79 and by F. (79) (or sirnply by F) the order filtration on 79. For a 
given rational number p/q > 0 we denote by Lp/q the filtration on 79 defined by 
pF + qV. The Lp/q-order of a rnononfial 9ttOOt is equal to p([/31 + k) + q(k -I), 
where I/1 =/ ÷ "' ÷/-, we will simply write L = L,,/q if no confusion arises. 
For p > 0 the associated graded ring grZ'(79) is canonically isomorphic to tf.OiT*ai 
[10, p. 407], where rr : T*A -+ A is the canonical projection and OIT-al denotes 
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holomorphic functions on T*A that are polynonfials on the fibers of 7r. In local 
coordinates, grL(/)0) is expressed as C{y}[r,y*, r*] where/)0 is the stalk of/) at 
the origin. 
Given a differential operator 

pky t 0 0 , 

the L-order of P is the maximun value of Æ(I/l + #) + q(k - l) over the monomials 
of P. For p > 0 we define the L-principal symbol of P by 

a *l "8 k 
oL(p) = pay (T)(y ) (-T) 

where the sum is taken over mononfials with maximal L-order. Thc L-principal 
symbol of P is an element of grL(/)) and then is a function on T*A. In the classical 
case, i.e., for L = F, grF(/)) is identified with C{x}[,...,] = C{y,t}[,i,r] and 
the F-principal symbol of P is simply dcnoted by aF(P) =  pytr]r  where 
the sure is taken for ]fil + k maximmn. 
To each left idem I C /) we dcnote by aL(I) the ideal of grL(/)) generated by 
the set of O "L (P) for P G I. 
For each L-filtration on/) we associate a "good" L-filtration on M, by means 
of a finite presentation. The associated grL(/))-module grL(M) is coherent (sec 
[10, 3.2.2]). The radical of the mmihilating idem Anng(Vx)(grL(M)), which is 
independent of the "good" filtration on BI, defines an analytic subvariety of T*A. 
This variety is called the L-characteristic variety of M and it is denoted by ChL(III). 
Suppose now that Z C X is a smooth hypersurface transverse to Y. Suppose 
for simplicity that Z is defined in local coordinates by Yl = 0. The conormal space 
A' := TrzZ is a smooth subvariety of A = T.X defined in local coordinates 
by Yl = 0. So T,A is the subvariety of T*A defined in local coordinates by 
Yl = Y'* = r* = 0, where f = (y2 .... , y_). 

Definition 2.1 ([9], [12]). We say that Z is non-micro-characteristic of type L 
for M if T,A çl Ch(M) is contained in TA. Sometimes we will say that, if this 
condition holds, Z is non-L-micro-characteristic for M. 

The sheaf of rings grL(/)) is endowed with two graduations: The first is induced 
by the F-filtration and the second is induced by the V-filtration. Recall Laurent's 
definition of slope of a coherent/)-module M. 

Definition 2.2 ([10]). The rational number -p/q is said to be a slope of BI with 
respect to Y at the origin if and only if the radical of the idem Anngr(Zx.)(grL(/ll)) 
is hot bihomogeneous for F- nor V-graduations. 

An important consequence of the work [11] (see Théorème 2.4.2) is what follows: 
A holonomic/)x-module AI is regular with respect to Y at the origin if and only 
if M bas no slope with respect to 1" at the origin. We will use this fact freely in 
the text. 

Remark. In [12], ce and 0 (F and V) were included in the set of the slopes. We do 
hot include them in the set of the slopes in this paper. 
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Finally, the following result by Laurent and Mebkhout allows induction on the 
number of variables to calculate slopes. 

Theorem 2.3 ([12, Corollaire 2.2.9]). Let Iii be a holonomic 79x-module. Let 
Z and Y be transverse smooth hypersurfaces on X such that Z is non-L-mzcro- 
characteristic (for ail p > O, q > O) for AI. Then the slopes of Iii with respect fo Y 
equal the slopes of AI' with respect to Y' = Z ç3 Y, where iii' is the restriction of 
1I fo Z. 

This is a deep result in/)-module theory. Its proof uses the algebraic-analytic 
comparison theorem ([11, Theorem 2.4.2]) and a Cauchy-Kowalewska theorem for 
Gevrey functions with respect to Z ([12, Corollaire 2.2.4]; see also [14, Théorème 
6.3.4]). 

3. COMPUTING SLOPES BY REDUCING THE NUMBER OF VARIABLES 

We have introduced the notion of the slopes and the invariance of them under 
restrictions satisfying a condition on L-characteristic varieties. In the sequel, we 
assume that out ideal is that of the Weyl algebra An. Constructions in sheaves such 
as restrictions and L-characteristic varieties in the previous section can be done via 
constructions in the Weyl algebra as we usually see in the computational 79-module 
theory. 
We are interested in computation of the slopes. The slopes of An/I (at the 
origin) along xn = 0 can be computed by the ACG algorithm introduced in [2]. 
In this section, translating Laurent and Mebkhout's result into computer algebra 
algorithms, we will give a preprocessing method for the ACG algorithm to accel- 
erate the original. The preprocessing is useful for a class of inputs including GKZ 
hypergeometric ideals as we will see in Section 4. Let us first recall the ACG 
algorithm. 
A weight vector is an element W = (Ul,...,un,vl ..... vn) E R 2n such that 
ui + v > 0 for ail i. This weight vector IV = (u, v) induces a natural filtration 
on A,, and it is called the W-filtration. The associated graded ring is denoted by 
grW(An), and for each left ideal I C A, the associated graded ideal is denoted 
by inw(I) or in(,v)(I). Hcre, the initial ideal in(,v)(I) is the ideal generated by 
in(,v)(f), f E I in grW(A,). When ui +vi > 0, it is an ideal in the polynomial ring 
of 2n variables: grW(An) = C[Xl,...,xn, çl,...,çn]. The initial ideal of I with 
respect to the weight (u, v) is generated by the (u, v)-initial terres of a Gr6bner 
basis of I by an order that refines the partial order defined by (u, v). See, e.g., [17, 
Theorem 1.1.6]. 
Consider the filtration L = pF + qV, p > 0, q > 0 introduced in the previous 
section. The ideal aL(I), which gives the L-characteristic variety, can be expressed 
in terres of the initial ideal as follows: 

g = p(0,...,0, + q(0,... ,0,-1,. 
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For two weight vectors W and W' and a terre order <, we denote by <w,w, the 
order 
X«ôI <W,W' xaôb 
oe W. ( c ,13 ) < W. ( a, b ) 
or W-(a,/3) = W. (a,b) and W'-(a,/3) < W'-(a,b) 
or W*  (a,/) = W* - (a, b) for both W* = W, W' and xc9  < xaE b. 
To each differential operator P = pZxc9  6 A, we associate the Newton 
polygon N(P) of P (with respect to x = 0) defined as the convex hull of the 
subset of Z 2, 
U (11,.- .) + (-N) 2. 
p¢0 
Let I be a left ideal in A. As we said in Definition 2.2, the notion of slope of 
differential system was introduced by Y. Laurent [10]. Let us give here a slightly 
different but equivalent definition: the number r, -oo < r < 0, is a geometric 
slope of I (or of A,/I) with respect to x, = 0 if and only if /Cr(-T)F+V(I) is hot 
bihomogeneous with respect to the weight vectors F = (0,..., 0, 1,..., 1) and V = 
(0,..., 0,- 1,0,..., 0, 1). Following [2], we say that the number r, -oo < r < 0 is 
an algebraic slope of I (or of A,/I) if and only if er (-T)F+v (I) is hot bihomogeneous 
with respect to the weight vectors F and V. The geometric slope is simply called 
the slope in this paper if confusion does not arise. Note that we may consider inL (I) 
instead of cr(I) so long as we are concerned about homogeneity. For algebraic or 
geometric slope r, the weight vector L = (-r)F + V lies on a face of the GrSbner 
fan of I ([3], [17]), which yields the following algorithm. 
Algorithm 3.1 ([2], ACG algorithm). 
Input: G = {PI,.-., P,} (generators of an ideal I). 
Output: Al! algebraic and geometric slopes of A,/I with respect to x, = 0 at the 
origin. 
geometric_slope = O; algebraic_slope = O; 
r= (0,...,0, 1,..., ); V= (0,...,0,-,0, .... 0,); 
= 1; q = 0; slope ---- -oo; prev±ous_slope -- slope; 
while (slope! = O) { 
L = pF + qV; 
G ---- a r6bner basis of I with respect to the order <L,V; 
slope = the mininlum of 0 and 
{the slopes r of the Newton polygon N(P) I P 6 G, r > previous_slope} 
if slope -- O, then return (algebraic_slope and geometric_slope). 
if crL(G) is not homogeneous for F nor V then { 
algebraic_slope = algebraic_slope U {slope} 
} 
if V/G) is not homogeneous for F nor V then { 
geometric_slope = geometric_slope U {slope} 
} 
p ---- numerator(Islopel); q ---- denominator(Islopel); 
previous_slope = slope; 
return(algebraic_slope and geometric_slope) 
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V 

FIGURE 1 

Here, we use the COlvention F := ocF + 1". Note that the ideal J = (fl,..-, 
C C[xl,..., :r,, 1,... ,(,] is holnogeneous for the weight (u, v) E Z 2' if and only 
if all (u, v)-hollogelmoUS subsulnS of ri belong to the idem J. 
For ordinary differential equations, the ACG algorithm is nothing but the well- 
known Newton polygon method; see two exaluples below. 
Exalnple 3.2. Put G = {xï+10 + p}, n = 1, p E N. Then, the ACG algorithm 
returns {-p}. (exp(x -) is a classical solution of G.) 
Example 3.3. Put G = {2x(xa01)  +xîO + 1}, n = 1. Then, the ACG algorithm 
returus {-1/2}. ( exp(x[ 1/2) is a classical solution of G.) 
The next example of two variables is generated by the computer algebra system 
kan/k0 [18]. 
Example 3.4. We consider the GKZ hypergeometric ideal I associated to the 
matrix A = (1,3) and/ = -3. We will compute the slopes of A2/I at the origin 
along x2 = 0 by the ACG algorithm. The GrO)ner basis of I for the weight 
(0, 0, 1 , 1 )is 
[ xl*Dxl+3*x2*Dx2+3, -Dxl-3+Dx2, -3*x2*Dxl^2*Dx2-5*Dxl^2-xl*Dx2, 
-9.x2 2*Dxl*Dx2- 2-36*x2*Dxl*Dx2+x1-2*Dx2-20*Dxl, 
-27.x2- 3*Dx2- 3-189.x2- 2*Dx2- 2-xl ^ 3*Dx2-276*x2*Dx2-60 ] 
Here, Dxi and xi stand for Oi and xi, respectively. The Newton polygons N(P)'s 
are given in Figure 1. 
Prom the Newton polygons, -1/2 is the candidate of the first slope. Next, we 
compute the Gr6bner basis for the weight (0, -2, 1, 3) = (0, 0, 1, 1) + 2(0, -1,0, 1). 
The Gr6bner basis is 
[ xl*Dxl+3*x2*Dx2+3, Dx2-Dxl'3 ] 
The radical is generated by 
[ -xl*Dxl-3*x2*Dx2, -Dxl^3+Dx2, 3*x2*Dxl'2*Dx2+xl*Dx2, 
-9,x2-2,Dxl,Dx2^2+xl-2,Dx2, 27*x2«3*Dx2-3+xl-3*Dx2 ] 
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Here, xl = Yl, x2 = vE*, Dxl = g, Dx2 = --7-. Itis not bihomogeneous, and then 
-1/2 is a geometric and algebraic slope. 
By looking at the two Newton polygons of xl*Dxl+3*x2*Dx2+3, Dx2-Dxl-3, we 
see that there are no slopes larger than -1/2. Then, the ACG algorithm terminates 
here. 

The ACG algorithm requires a repetition of Gr6bner basis computations in the 
Weyl algebra of 2n variables to evaluate the slopes of A/I. However, if xi = 0 
is non-micro-characteristic of type L = (-r)F + V for all -oe < r < 0 and the 
restriction of A/I to xi = 0 is singly generated, we can preprocess the input so 
that the input idem for the ACG algorithm lies in the Weyl algebra of 2(n - 1) 
variables. The correctness of the following algorithm can be shown by Laurent and 
Mebkhout's Theorem 2.3. 

Algorithm 3.5 (Computing slopes with a preprocessing). 
Step 1: Check if xi = 0 is non-micro-characteristic of A/I for all types L by 
calling Algorithm 3.6. 
Step 2: Compute the restriction of .4,/I to xi = 0 and check if it is expressed 
as D/I  where D  is the Weyl algebra of 2(n - 1) variat)les. 
Step 3: If we failed eithcr in Step 1 or in Step 2, then apply the ACG algorithm 
for I. 
If we succeeded both in Step 1 and in Step 2, then try to reduce more variables 
or apply the ACG algortih,n for I . 
We tan compute restrictions of a given Z)-inodule by using Oaku's algorithm [15]. 
This algorithm is ilnplemented in computer algebraic systelns Macaulay2 and Kan 
[7], [18], [13]. Therefore, the remaining algorithmic question for the preprocessiug 
is to determine the range of type L = (-r)F + V for which x = 0 (i _< n - 1) is 
non-micro-characteristic. It follows from the definition of non-micro-characteristic 
that the question is nothing but to find the segment (-oc, r) such that 
for r G (-0% r). Here, 12(f .... , fro) is the affine variety defined by the polyno- 
mials f,..., f,. The inclusion condition can be algebraically rephrased as 
Since the Gr6bner fan is a finite union of Gr6bner cones [2], the range can be 
determined by a similar method with the ACG algorithm. 
Algorithm 3.6. 
range_oI_nonC(H, r0) 
Input: H is a finite set in A,, r0 is a negative number or 
Output: r such that x = 0 is non-micro-characteristic of type (-r)F + V 
for A/A . {H}, for r Ç [ro, rl). 
previous_slope = slope = ro; 
while (slope! = O) { 
p = numerator(Islope]); q = denominator(Islope]); 
L = pF + qV; 
 = a Gr6bner basis of H with respect to <L; 
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"i:f k/(O'L(H),Vl,V,...,V_I,T *)  V and 
o'PF+(q+e)V(H), Yl, Y,-  , n--1, T*)  1 
then { 
previous_slope = slope; 
slope = the minimum of 0 and 
(the slopes r of the Newton polygon N(P)Pe G, r >previous_slope} 
return(slope); 
} 
return(0); 
ttcre, « is a sufficicntly small positive rational number so that pF + (q + «)V lies 
in the interior of a GrSbncr conc. 

In the case when r0 = -ec, we use the convention numerator(]r0[) = 1 and 
¢lenominator(]r0]) = 0. and F-non-micro-characteristic means that it is non-char- 
acteristic in the classical sense. 

Example 3.7. Sui,pose n > 2. Then range_ofnonNC({0î - 02},-oe) returns 0. 

If the function range_ofmonMC(l,-oe) returns 0, then X 1 = 0 is non-micro- 
characteristic of type pL + q" (p > 0, q >_ 0) for A,,/I. We note that it is not 
always a clever strategy to call the function with the full set of generators. In 
fact, if Xl = 0 is non-lnicro-characteristic of type L for An/J, then it is non- 
nficro-characteristic of type L for An/I for any I _D J. Therefore, for Step 1, it 
is sometimes more efficient to call the function range_ofmonl'IC for a subset of the 
generators of the input idem as we will sec in the case of GKZ hypergeometric ideals 
in the next section. 

 1. COMPUTING SLOPES OF :-(1,a2 ..... a,)(/3) 
Put A = (1, a2,..., an), al  1 < a2 < -.- < an. We will evaluate the slopes of 
the GKZ hypergeometric ideal HA(C) associated to the 1 x n matrix A and/3 E C 
by using the general algorithm given in Section 3. To apply this algorithm, we need 
to find non-nficro-characteristic varieties and compute the restrictions of 7-/A(/3) to 
these varieties. When fl,--.,f, are polynomials in C[xl,...,xn,çl,...,çn], we 
denote by 12(fl,..., f,) the affine subvariety in C 2n defined by the ri. 
The following tlleorem can be shown by a standard method of Koszul complex 
([11, [61)- 

Theorem 4.1. The characteristic variety of 7-/.4(/3) is ")(Ç1 .... , Çn-1, Xnn). In 
particular, the singular locus of 7-/.4(/3) is Xn = O. 

Note that there is no slope along xi = 0, 1 _< i <_ n - 1, which can be shown 
easily. 
Recall that F = (0,...,0,1,...,1) and V = (0,...,0,-1,0,...,0,1). For a 
positive number p and a non-negative number q, we define the weight vector L = 
pF + qV. In Section 2, we explained the notion of non-micro-characteristic. "When 
the variety is yi = xi = 0, this notion is rephrased as follows: for a given left 
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A,¢module A,/I, the hyperplane Yi = 0 (1 _< i _< r - 1) is called non-micro- 
characteristic of type L when 
Proposition 4.2. For the hypergeometric A,-module  A (fl), te vamety Yi = 0 
(1 < i < n -- 2) is non-micro-characteristic of type L for ail L = pF + ql , p > O. 
Pwof. Consider 0ï -0'  HA(ff). For all L aud for i < j 5 n- 1, we bave 
[Yi I , which implies that yi 0 is non-micro-charteristic of 
type L.  

Now, let. us apply the second step of the algorithm to evaluate the slopes, i.e., 
we will compute the restriction of 7-/A(fl) to y, = xi = 0 (1 < i _<  -- 2). Let s be 
an indeterminate. Consider the ideal HA[SI in A,[s] generated by AO - s and IA. 
Theorem 4.3. We bave a le D'[si-module isomorphism 
(4.1) A[s]/(A,,[s]HA[s] + xiA[s])  D'[s]/D'[s]Hv[s], i % 1. 
Here, D' = C(x,..., xi-1, xi+,,..., x, 01,.... 0i--1,0i+1 ..... On} and A' = (1, 
    ai-l i+1 .... , an). 
Pro@ Fix an order  such that 0, > 0, > O._ > -.- > 0. Then, 
0n --0Ï n' 0n--1 -- )an-ll ' "'" "02 -- 0Î 2 
is the reduced GrSbner basis of IA with respect to >. 
For each i (2  i  ), by applying the smne method as in the proof of [17, 
Theorem 3.1.3], we can prove that in(_«,,«,)(HA(s)) is generated by 40 - s and 
i,,(Oj -aï) (j = 2 .... ,n). 
Define the -filtration F[s] of A,[s] by 
 compute the restriction of A[s]/H(s) to z = 0 by Oaku's algorithm (see, 
" -xO ) xO, 
e.g., [17, Theorem 5.2.6, Algorithm 5.2.8]). Since m(_«,,«,)(x0 ' ' = 
the b-function of HA(S) along x = 0 is b(p) = p. Therefore, by [17, Theorem 5.2.6], 
we bave the following isomorphism of leff D'[si-modules: 

A,[s]/(HA(S) + xAn[s]) 
_ Fo[sl/ Fo[sl(AO- s)+  
j=2,j¢i 
_ D'[sl/ (D'[s](A0- s) + 

Fo[8](Oj - Oï) + F_l[8](Oi - Oî') + XiFl[8]) 

We can specialize s to any complex number /3. In fact, we have the following 
theorem. 

Theorem 4.4. 
A,/(A,I-IA(/3) + xiA,) _ D'/D'HA,(I), i ¢ 1. 
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Proof. Put L = ciO1 + a2x202 +... + annOn -- /. We note that [ôk -- Oî k, L] = 
ak(Ok -- ôî). Fix a lexi«ograI)hic order 
0_  --.  0. Then, we can show that 
{L,O. " "" 
--01 , 0n-1 -- 1 , .... 02 -- aï } 
is a Gr6bner basis of H() with respect to the order (_,,,) by Buchberger's 
S-pair criterion. In fact, we have 
p(O - o, o + o +... + o. - ) = r(o» - oï) - (o - o )r  0. 
The remaining checks are easy. Thercforc, we con«lude that in(_,,)(HA(ç)) is 
generatcd by AO - ç and inc, (Oj - 0ï ) (j = 2,..., ,). The rest of the proof is the 
saine as that of Theorem 4.3. 
The thcorcm mcans that thc restriction of «() to x  0 can be exa«tly 
cxprçssed in terres of thc GKZ systcm for snmllcr A. By applying our Algorithm 
3.5 for computing slopes by rcduction of the nmnber of thc variables to the variables 
x2,..., x_2, wç ohtain tlm following tlmorcm from Proposition 4.2 and Theorem 
4.4. 
Theorem 4.5. The geometric ôlopes of 4() along x  0 at the origin and 
(,«,_,«)(ç) along xa  0 af the origin coincide. 
Example 4.6.  note that (thç alsçbraic slopes)  (the geonmtric slopes) in 
gencral. For example, let us apply thc ACG algorithm to get the algebraic slop 
of HA(--30) for A  (1, 3.7). This idçal is generated by 
The output is 
(-,-a/4,- /). 
On the other hand, if we apply the AÇG algorithm to get the geometric slopes, the 
output is (-3/4). 
np 4.7. (t l f (,«_,«)())  (th l« f u(,«)()) i - 
Let us take the exmnple: A  (1,3.7). Considcr the hypergeometric Aa-module 
Aa/I where I  H(,a.7)(-30). As we have seçn, the slope of this svstem along 
x = 0 is (-a/4). 
Çonsider inL(I) for L  F + 4I . By computin8 the Gr5bner bis with respect 
to L, we can see that 
ç(i.(I)) = 
It is not included in ç(ç2). Hçnce x  0 is micrcharactcristic of type L, and we 
cannot apply the restriction criterion. 
The condition "non-microcharacteristic for ail the filtration pF + qI" cannot be 
taken as a way to evaluate the slopes by the restriction. In fact, it can be easily 
checked by the AÇG algorithm that the set of the geometric slopes of (,«)() is 
l t (/( - ). ,c, t t 
not equal to (-3/4). 
e bave shown that the computation of the slopes of A () is reduced to the 
case of three variables. The slopes in this case are as follows. 
Tho 4.s. (t l f (,«_,«)())= (_/(_- )). 
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Pro@ We fix some notation: 
(1) A=(1,a,b) EZ 3andl<a<b. 
(2) P1 = cgî - 02, P2 = cgï - 03, P3 ---- c92b -- C9, P4 = XlCgl -t- ax2c92 A- bx303 - . 
Let A be the linear form with slope -a/(b-a) (i.e., A = aF + (b-a)V). 
Let L, L' be linear forms. We say that L > L' if slope(L) > slope(L'). 
We will write gl = Xl, g2 = x2, t = x3. 
operators Pl, P'2, P3, P4 are in H = HA([3). Then, we have the following 

(3) 
(4) 
(5) 
The 
claires. 
(1) 
(2) 
(3) 
(4) 
(5) 

For all liuear forms L we have oL(p1) ----- (ç{)a and so r/ e aLrL for all 
L. 
For all linear forms L we have aL(p4) = yl? -- cty2] -- br*(-r). 
For all linear forms L > A we have aL(p3) = (r) « and so r 6 V/-H) for 
all L> A. 
Thus, for ail L > A we have ChL() C T=0CaUTC a, and then  
is bihomogeneous and L is hOt a geometric slope of . 
On the other hand, for L < A, we have aL(Pa) = (ç)b. Then *1 6 H) 
and ChL() C T;=0C a U TC a. So L is not a geometric slope of . 
(6) Therefore, the only possible geometric slope of  is A. 
Now, suppose that A is hOt a slope. Then, there is no slope, which implies that 
the L-charteristic variety chL(A()) is the saine for all L = pF + qV, p, q > 0 
by [3] and [10, Théorbme 3.4.1]. It follows froln 
[PI,2I = 0, [P1,P4I = oP1, [2, P41 = bP2 
and the Buchberger algorithm that {P1, , P4} is a Gr6bner basis for the order 
defined by the weight vector L = (0, 0, -N, 1, 1, N + 1), N  b and a tie-breaking 
terre order such that z  a    . Therefore, the initial ideal inL(HA(ç)) is 
generated by î, a, Zll + az + baa and hence the L-characteristic variety is 
equal to T=0Ca U TaC a. This fact contradicts that the L-characteristic variety 
is the saine for all L.  

5. I:IATIONAL SOLUTIONS AND IEDUCIBILITY 
Out ultimate aire in studying the slopes of HA (/) is to get a better understanding 
of solutions of this systeln. We are far from the goal, but to this end, it will be useful 
to present some facts on classical solutions and a relation to generalized confluent 
hypergeometric functions. 
In the case of the hypergeometric ideal associated to hornogeneous monomial 
curves, Çattani, D'Andrea, and Dickenstein [5] studied rational solutions and re- 
ducibility of the system. We will study rational solutions and reducibility of out 
system. 
Theorem 5.1. Any rational solution of the hypergeometric system H(1,« ..... 
is a polynomial. It bas a polynomial solution if and only if l  N = {0, 1,...}. The 
polynomial solution is the residue of exp ( xt «' ) t -¢ at the origin t = O: 
/ceXp(-.xita') t-[3. 
Here, C is a circle that encircles the origin in the positive direction. 
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Proof. Since the singular locus of HA(ff) (A = (al,a2,... ,an), a ---- 1) is x, = 0, 
any rational solution f i8 a Laurent polynomial with poles on x = 0. Take a weight 
vector u, = (0, 1.1,..., 1). Then. we bave in(IA) = (Oz,..., 0}. The initial terre 
in(f) is annihila.ted by 
Therefore, xï = inw(f). This irai)lies   N, because f h a pole only on xn = 0. 
Take a Z-bis of Ker (Z '  Z) as 
(-a2, 1,0,...,0), (-a3,0.1,0 ..... 0),..., (-a,0,...,0, 1) 
to construct series solutions. Since [17, Proposition 3.4.1] holds for non-homo- 
geneous A as well, a formal series solution g of H.4(fl) satisfying in(g) = xï can 
be mfiquely expressed  
(5.1)  #(ff- 1)-L:(- Ema + 1 ) x x. xï. 
When   N, it is a polynonfial. The rcst of the thcorem is easy to show. 

Let R be the ring of (tiffcrential ol)erators of 7 variables with rational function 
coefficients over k = C. A left idem d of R is called irreducible when d is a maximal 
idem in R. We will stu(ly the reducibility of R. HA(ff). 
We assume that J is zero-dimensional, i.e., r = dimc(x) R/d < +ec. Let V = 
I'(J) be the vector space of holomorphic solutions of d on a simply connected open 
set contained in the non-singular domain of J. It is known that dimc " = r. Define 
I(V) by R.{tR]tof=O for all f  V}. IfJ C I('),J : I(V), then we have 
dimc(x) R/d < dimc V = r because of the zero-dimensionality of d. Therefore, we 
have 
J = I(V(J)). 
Under this correspondence of ideals and solutions, a zero-dimensional idem J of R 
is reducible if and only if there exists a proper subspace W of the solution space of 
V(J) such that 0 < dimc(x) R/I(W) < dimc(x) R/J. In the case of one variable, 
the reducibility is equivalent to soEving that the generator of the idem tan be factored 
in R. 

Theorem 5.2. The system of differential equations R-H4(fl) is reducible if and 
only if fl  Z. 
Proof An)" curve is Cohen-Macaulay. By applying the theorem of Adolphson [1], 
the holonomic tank of HA(ff) is a, for all ff. 
Put M(ff) = A/HA(fl). Consider the left A-morphism 
(5.2) 0 : M(fl) M(fl+ 1). 
It h the inverse when fl ¢ -1. Therefore, M(-1)  M(-2)  M(-3) --. and 
M(0)  M(1)  M(2) ---. 
When  G N, the system admits a polynomial solution; then it is reducxble. It 
is also easy to see that when  G Z<0, the equation is reducible. In fact, consider 
the left D-morphism 
01 : 51(-1)  M(0). 
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It induces a morphism to the solutions by 

The solution f = 1 of M(0) is sent to zero. So the image of oqt gives a proper 
subspace of solutions in the solution space of M(-1). To find differential equations 
for the subspace, take all te such that te0 E HA(O). Then, {t e} C HA(-1). By the 
isomorphism (5.2), we conclude that when/3 E Z<0, the system is reducible. 
Let us prove that the systeln is irreducible when/3 ¢ Z by applying the result 
of Beukers, Brownawell, and Heckman [4]. For this purpose, we first construct a 
convergent series solution of HA(P). Take w = (1, 1 ..... 1,0). Then the degree of 
in(IA) is equal to an. Since IA contains the elements of the form/)î -/)', the 
radical of in (IA) is (cgt .... , c9,_ 1). Therefore, the top-dimensional standard pairs 
have the form (0 b, {n}). 
Let v be the zero of the indicial idem associated to (0 b, {}): 

Vl --- bi,  . . , Vn-1 = bn-t , Vn -- 

an 

Assume/3 ¢ Z or/3 » 0. Taking the lattice basis 

(a2,-1,0,...,0),..., (an, O,...,O,-1), 

we have the following an linearly independent convergent series solutions: 

(5.3) 

m(?N n-1 

ve(ve - 1)---(ve -- mm + 1)'"vn(vn - 1)'"(vn - mn + 1) 

(Vl -}- 1)(Vl A- 2) "'" (Vl A- ,akrak) 
.(ï)...(«x.,  ,   

We consider the change of variables: 

YI = Xl,Y2 = Xî2/X2, "'',yn = Xîn/Xn  

The inverse of this change of variables is also rational, and the change of variables 
induces that of cgi. We denote by  the operation of this change of variables 
of xi and c9. Since the irreducibility is invariant under any birational change of 
variables, we will prove the irreducibility of the ideal J = R. tP(x-HA(/3)x ) where 
n = c@(o,,,,..., o,,,.). 
Let V be the solution space of J spanned by the series @((5.3)-x -) near y = 0. 
If J = I(V) is reducible, then there exists a proper subspace W of J such that 0 
dimc() R/I(W) < a,» We consider the vector space 
W}. It is easy to see that dimc()R'/I(W') 
Let us prove that dimc W  = a, when/3 ¢ Z, which implies the irreduciblity of J 
by a contradiction. 
We restrict the series ((5.3)-x -) to yl = Xl = 0, y2 = xï/x2 = O, ..., 
«'-" = xï/xn by z. Without loss of generalitv, 
Yn-1 = X 1 ]X n = 0 and replace y, 
we may assume Vl = 0. Then the restricted series has the form 

(5.4) 
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It is annihilated by the ordinary differential operator 

(a,O)(a,fl- 1)...(a, Oz-an ÷ 1)- z(O ÷ vn). 

By replacing z/a" by x, we obtain the generalized hypergeometric ordinary differ- 
ential equation 

(5.5) 

0x(0x - l/a,0    (0x - (an - 1)/an) - :«(0 + vn). 

By [4], this ordinary differential equation of tank an is reducible if and only if 
vn - k/an  Z for ail k = 0, 1 .... , an - 1. If one of theln is ail integer,/3 becomes an 
integer. Therefore, the ideal I(IV') contains the principal ideal generated by (5.5), 
which is maximal when/3  Z. We conclude that I(IV') is generated by (5.5) and 
hence dimc II "' = an. [] 
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ASYMPTOTICS FOR LOGICAL LIMIT LAWS: 'VVHEN THE 
GROWTH OF THE COMPONENTS IS IN AN RT CLASS 

JASON P. BELL AND STANLEY N. BURRIS 

ABSTRACT. Compton's met hod of proving monadic second-order limit laws is 
based on analyzing the generating function of a class of finite structures. For 
applications of his deeper results we previously relied on asymptotics obtained 
using Cauchy's integral formula. In this paper we develop elementary tech- 
niques, based on a Tauberian theorem of Schur, that significantly extend the 
classes of structures for which we know that Compton's theory can be applied. 

1. INTRODUCTION 
We are primarily interested iii })eillg able to show that ail answer exists to the 
following question: given a class K of finite relational structures and a property : 
What is the probability that a finite structure, randomly sclected 
from/C, has thc propcrty 
While pursuing this goal we are able to prove (in Corollary 4.3) the conjecture 
([10], p. 462) of Durrett, Gra, lovsky and Gueron arising in the study of coagulation- 
fragmentation processes which says that if S(x) =  s()x  is a power series with 
positive coefiïcients such that s(7 - 1)/s(n)  p, then the coefficients of the power 
series expansion of exp (S(x)) bave the saine property. 
The notion of probability t.hat we use is to take the proportion q of finite 
structures of size  in/C (we only count up to isomorphism) that have the property 
(I), and then to take the limit q of the sequence q as  goes to infinit.y. This limit, 
when it exists, is also called the asymptotic denszty of the class of members of/C 
that satisfy (I). 
It has been known since the raid 1970s that a few well-known classes, like graphs 
or directed graphs, are such that if (I) is a property defined by a first-order sentence, 
then the probability must exist, and be 0 or 1. If/C is a class such that every sentence 
in a given (logic) language L defines a property for which a probability exists, then 
we say/C has an L-limit law. 
In the 1980s Kevin Compton gave a new method for proving that a class /C 
of relational structures has a monadic second-order linfit law, a method that only 
depends on analyzing the growth rate of a(n), the number of structures of size n 
in/C. This applies to classes that are closed under disjoint union and components. 
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For such classes the count function p(n), the number of components of size n in 
KT, is often nmre readily available than a(n), and we would like to know conditions 
on p(n) that guarantee a logical limit law. The best results of this type previously 
known were 
 if p(n) = O(nC), that is, p(n) is polynomially bounded, then /C has a 
monadic second-ortier 0-1 law (Bell [1]), and 
 ifp() = Cil ' + O(7'), where 0 < 7 < fl and C > 0, then /C hasa 
nmnadic second-order limit law (see [4], Chapters 5 and 6). 
We will nmke use of a Taul)erian theorem of Schur to extend these results to 
cover a large number of new classes of structures. For example, we will show that 
p(n) ,' anbe cnd, 
with d < 1 and a, c > 0, yields a monadic second-order 0-1 law; and 
Corollary 9.4. For # < 1 < fl and C > 0, or p = 1 </3 and C > l, the condition 
p(n) - Cn/-" 
guarantees that E has a monadic second-ortier limit law. 
In addition to the standard "big O' and "'little o" notation from asymptotics we 
use the following: 
notation means 

f(n) --< g(n) f(n) is eventually less than g(n) 
f(n)  g(n) f(n) is eventually less than or equal to g(n) 
f(n) >- g(n) f(n) is eventually greater than g(n) 
f(n)  g(n) f(n) is eventually greater than or equal to g(n) 
As mathematics symbols we will be using upper case boldface roman letters 
exclusively to denote power series, and the corresponding lower case ordinary italic 
letters naine the coeflïcients. Thus we will use A(x) = -na(n)x n .... , T(x) = 
E, t(n)x ' . 
2. THE CLASS RTp 
The sequences in RTp play a central role in the study of Compton's development 
of logical limit laws. We find conditions to guarantee membership in this class and 
then apply them to prove logical limit laws. 
Definition 2.1. RTp is the collection of sequences s(n) of real numbers that satisfy 
(a) s(n) - O, and 
(b) lira s(n-1) _ P- 
- 
We also say that a power series S(x) is in RTp if its sequence of coefficients s(n) is 
in RTp. Also, if f(x) is a function that admits a power series expansion  s(n)x n 
about 0, then we say f(x) is in 
The following lemma and corollary give the most bic information about the 
growth rate of members of 
Lemma 2.2. Ifs(n)  Tp with 0 < p < oe then, forO <  < p-, there is an N 
such that, for n  N, 
(p-'-) < () < (p-' +). 
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Prom this we see that a snmller p leads to nmch faster-growing sequences. 
Corollary 2.3. Ifs(n) E RTo and t(n)  RT with 0 < r < T < oe, then 
t(n) = o(s(,,,)). 
The class RT v has some remarkable similarities to the class RV, the class of 
functions of regular variation at infiuity with index a, but this connection does not 
seem to have been thoroughly researched. For 0 < p < oe the sequence p-n is 
perhaps the sinlplest member of the class RTv, and this sequence, along with RT1, 
completely determines RTe, since one can easily check that 
s(n)  RTp  s(n)p   RT1. 
II terres of power series this would be written as 
(2.1) S(x) G RTo ¢== S(px) G RTe. 
Here are three of the simplest examples from RTI: 
s(n) = c forc>0, 
s(n) = n c for c any real nulnber, 
s(n) = d'« forc<l<d. 
We can use these examples, with Proposition 2.4, to make a substantial collection 
of sequences in RTç. Multiplication of the sequences s(n) and t(n) gives s(n)- t(n), 
and division gives s(n)/t(n), where we define s(n)/t(n) to be 0 whenever t(n) = O. 
Proposition 2.4. RT is closed under multiplication, division, and asymptotically 
equal. For 0 < r,- < oe, f s(n)  RTe and t()  RT.,-, then 1/s(n)  RT1/«, 
and s(n).t(n) E RTo.,-. Furthermore, S(x) E RT v iff S'(x) E RTp iff xS(x) E RTp. 
Pro@ (Straightforward) [] 
With this we can easily see, for exmnple, that for a, c, p > 0 and b any rem 
number, if s(n) ,, anbecoE/p , then s(n)  Tp. Just such an example played 
an important role in finding the first applications of Compton's 1989 theoretical 
development of logical limit laws. A key fact about this particular sequence is that 
anbe cOE  T1, and it is eventually nondecreasing. 

3. THE CAUCHY PRODUCT 
The Cauchy product R(x) = S(x)-T(x) is defined by 
(n) =  s(). t(n - ). 
k=0 
The following two lemmas and corollary help us extract information about RT 
classes from the Cauchy product. We start with the classic Tauberian theorem of 
Schur. 1 
Lemma 3.1 (Schur). With 0 < p < oc, suppose that 
(a) A()  ev, 

lit is a Tauberian theorem since it gives an extra condition, namely the radius of convergence 
of B(x) > p, to allow us to go from a generalized limit, lim C(x)/A(x), to an ordinary limit, 
lim «(ri)la(n). 
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(b) B(x) bas radius of convergence greater than p, and 
(c) B(p) > 0. 
Let C(x) = A(x)-B(x). Thon 
c(n) -- B(p)- 
Proof. (See B«nder [3] or Burris [4].) 

Thus frOlll the hypotheses of Schur's Lemma we deduce that C(x) E RTp. 
Corollary 3.2. Witb 0 < p < oc, suppose A(x) E RTp and tbe radzus of conver- 
gence of B(x) is great«r than p. If B(p) > 0. then 
A(x)-B(x) E RTe. 
Proof. Let C(x) = A(x)  B(x). Prom Schur's Lemma we have 
c(n) ,'-, B(p)-a(.,); 
so, by Proposition 2.4, C(x) E RTp. [] 

The next lenuna offers a variation Oll this thelne. 

Lemma 3.3. Stppose C(x) = A(x).B(x), where A(x) and B(x) bave nonnegative 
coeJficients and B(x) is zot tbe zero power series. If 
(a) A(x) ERTp, wh«r«O<p< 
(b) b(n) = o(c(n)), and 
c(.- 1) 
(c) 4 p, 
c() 
then C(x) E RT o. 
Proof. Froln the following equivalent statements, 
A(x) = B(x)-C(x) iff A(px) = B(px)-C(px), 

it suffices to prove the lemlna in the case that p = 1. With p = 1 the goal is: 
from (a') A(x) E RTe, (b') b(n) = o(c(.)), and (c') 
prove that C(x) E RTe. 
Let e > 0. By (a ) there exists an integer _AI such that, for n > _AI, 
(3.1) la(n) - a(n- 1)1 < 
Since A(x) E RT implies a(n) - O, and B(x) is hot the zero power series, we have 
c(n) - O. So, in view of (c') we can also assume that _AI is sufiïciently large to 
guarantee 
{ c(n.) > 0, and 
(3.2) n >_ _AI ==> c(n-1) < 1. 
By assumption (b') we can find an integer N > _AI such that 
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(3.3) b(n) _ 
(]il + 1)max (1, a(0) .... ,a(M)) 
forn> N-M. 
Now suppose n _> 5I + N. Then, by (3.2), 
(3.4) 0 < «(n-Iii) < ... < «(n-1) 
For n >_ M + N, 

<_ c(). 

c(n) - c(n - 1) 

 --I 
= Ea(n-i)'b(i) - Ea(n- 1-i).b(i) 
= Z (a(n-i)-a(n-l-i)).b(i) 
i<n--hl 

<_ ec(n) + 
= 
Therefore, 1 - 2e <_ 
lira c(n-1) = 1, andsoC(x) E RT1. 
- c() 

n--1 
z=n--hl 

a(n - 1 - i) . b(i) 

ec(i)/(M+ 1) by (3.1),(3.3) 

ec(n)/(iiI + 1) by (3.4) 

c(n - 1) < 1. Since this holds for any n > Iii + N, we have 

4. EXPONENTIATION 

We will be particularly interested in knowing that exponentiation of a power 
series preserves membership in RTp. First we prove a special case of this result. 

Lemma 4.1. Let T(x) = exp (S(x)), where 
(a) S(x) E RT1, 
(b) the s(n) are nonnegative, and 
t() 
(c) lim inf -- C > O. 
 t(n - 1) 

Then T(x) E RT1. 

Proof. Note that C <_ 1 since the radius of convergence of T(x) is 1. From S(x) E 
RT 1 we know that xS'(x) E RT1. So, given  > O, we can choose Iii such that 
ms(m) - (m- 1)s(m- 1)1 < ems(m) 
t(n) C for rn > Iii. Also, we can choose N such that t(n - 1) > - for n > N. Froln this 
we see that 

t() c  
(4.1) n- r > N == > 
t( - r) 2  
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Differentiating T(x) = exp (S(x)) we have 
T'(z) = S'(x)T(x), 
and equating the coefficients of x -1 on both sides of this equation gives 
(4.2) nf(n) = E t(m) . (n - m)s(n- m). 
From this it follows that if  > N + M (we adopt the convention that s(m)  0 for 
m < 0), 
],,t(n) - (- 1)t(- 
  t(). ( - )(- ) 
O(m(n--I 
+  t(m).](n-m)s(n-v) - (n-l-m)s(-l- 
 () + . ( x ,()).  ()  (.) 
0<<1 
= ( 
Thus rit(n)  T; so t(n)  T1. 
L¢ 4.2. s() z a  p(s()) z al. 
there exists N such that s(n) > 2 - for n > N. Define (n) o be 1 for 0  n  N 
(n)  2-/n, foroE1, 
(x) + log(1-z/2) is a power series with nonnegaive coecients. Thus, for n  0, 
 0; 

SO 

lim inf  '(n ) 

_> 1/2 >0. 

To finish the proof, observe 
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Next we see that RTp classes are closed under exponentiation of power series, 
proving the conjecture ([10], p. 462) of Durrett, Granovsky and Gueron. 

Corollary 4.3. For 0 < p < oe, 
S(x)  RTp  exp (S(x))  RTp. 
Proof. This follows from (2.1) and Lemma 4.2. 

If we exponentiate a power series in RTp that diverges at p, then we obtain a 
power series whose coefficients grow much faster than those of the original series. 
Lemma 4.4. Let T(x) = exp (S(x)) where 
(a) S(x)RTp with O < p < oe, and 
(b) S() = . 
Thon s(n) = o(t(n)). 

Proof. First we prove this lemma for the case that the coeflïcients of S(x) are 
nonnegative. Choose N OE 1 and observe that, for n >_ N, 
N 
3>0 1- >_ l [xn]S(x)2 > l k:O 
t(n) 
. j.[ x]s(xP  -  
s(lç)s(Tl-- lç) 

and thus 

N 
t(n) > 1E s(k) s(n - k) 
s(n) -  k=O -(-) " 
Taking the lira inf of both sides, using the fact that S(x)  RTp gives 
t(n) 1 N 
li222f () >   (). 
k----0 
Now use the fact that S(p) = oe. 
For the general case, where some of the coeflïcients of S(x) can be negative, let 
p(x) be a polynomial such that (x) = S(x) + p(x) has nonnegative coeflïcients. 
Clearly (x)  RTp and (p) : oe. Then (x) = exp ((x)) is such that 
(4.3) (n) : o(î(n)), 
by the first part of the proof. By Corollary 4.3, (x)  RTp. Since 

T(x) 
by Schur's Lemma we have 
t() 

= exp (- p(x))- (x), 

 exp ( - p(p))  '(n). 

So from (4.3) we have s(n) = o(t(n)) since s(n) eventually equals '(n). 

A modest growth condition on the coefficients of S(x) guarantees that the co- 
efficients of exp (S(x)) satisfy a growth condition used to prove logical limit laws. 
But first we prove a technical lemma on membership in RTe. 

Lemma 4.5. Suppose S(x)  RTI. Then 
liminf s(n) > 1 

s(n)- 1  RT1. 
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Proof. Choose C, N > 1 such that s(n) > C, for n > N. Then, for n > N, 

(4.4) 
Now 

1 C-1 
1--- > > C-1 > O. 
s(.)- s() - 

(,-1) 1 
s(n- ])- ] _ l = () 
(n)-   () ' 
and the right side tends to 0 as n tends to infinity since s(n) E RT1 says the 
numerator tends to 0, and (4.4) says the dcnominator is bounded away from 0. 
Thus, 
lira s(n-1)-I = 1. 

Now we proceed with the analysis of the growth rate of the coefiïcients after 
exponcntiation. 
Lemma 4.6. Let T(x) = exp (S(x)) where 
(a) S(x) ERTp with O < p < oe, and 
(b) liminfs(n)p  > 1. 
Then T(x)  RTp and t(n - 1) 
t(,) '< p" 
Proof. Corollary 4.3 gives T(x) E RTp. To verify t(n - 1) - t(n)p note that 
t(n)p " - t(n - i)p'- = 
= [<o((0) + E ((..»  - 1/+ 9. 

Since S(x)  RTe, 

ns(n)p   RTe, 
and then Lemma 4.5 gives ns(n)p n - 1 E RT:[. Using Proposition 2.4 we have 

(4.5) s(n)p ' - lin E RT,. 
Choose N _> 1 such that, for n > N, 
(4.6) ns(n)p ' > 1, 
and let 

R(,) =  x +  ((,)- v,O«. 

By (4.5) we know that R(x)  RTe; so Lemma 4.2 gives 
(4.7) exp (R(x)) e RT]. 
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Noting that p(x) is a polyuoinial we have 
= [xn](exp(p(x)).exp(R(x))) 
 exp(p(1)). [x]exp(R(x)) 
bç (4.7) and Schur's Lemma, and thus, since t.he coefficients of R(x) are positive, 
t(,)p  - t(,- )p- » O. 

t( - ) 
This says t(n) -4 p [] 

5. THE STAR TRANSFORMATION 
Now we introduce a transformatioi on a power series S(c) that ploEvs an impor- 
tant ro]e in combinatorics. 
Definition 5.1. Let S*(x) =  s*()x n be the power series defined by 
*(o) = o, 
s*(,) = Z s(j)/k for, >_ 1. 
j k-=n 
Lemma 5.2. For 0 < p < l, zf S(x) E RTp bas nonnegative coe]ficients, then 
(a) se(n)  s(n), and 
(b) S*(z) E RTp. 
Proof. Since s(n) E RTp we know from Corollary 2.3 that 0 </3 < p- < c implies 
n2/3 " = o(s(n)) and s(n) = o(c ") 
since n2/3 n  RTe/B and an  RTl/a. Choose a satisfying p-i < a < /9-2 and 
choose C such that Is(n)l < Co n for all n. Choose N such that s(n) >_ 0 for 
n >_ N. Then 

ns*(n) = Z ds(d) 

<_ ns(n) + Z ds(d) 

= () + o(. "/) 
= ns(n) + o(s(n)) (since  < D-l). 
Hence ns*(n) ,. ns(n) since 0 <_ s(n) <_ s*(n), and thus s*(n) ,. s(n), giving (a). 
Then from Proposition 2.4 we have S*(x) E RTp. [] 
This result is best possible since one can easily find examples with p = 1 such 
that S(x) E RT but S*(x)  RTe. For example, S(x) = -x n E RTe, but 
S*(x) = -n cr(n)xn/n  RT1. However, the power series expallSiOn of -ff---x -S*'(x) 
1--x 
is much better behaved in this situation. 
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Lemma 5.3. 

Proof. Let S*(n) 

(5.1) 

since 

Also, 

(5.2) 

Suppose s(n) E RT1 is a sequence of nonnegatzve terres. 
s*(1) +-..+ 7/s*(n) E RT,. 
= s*(1) +--. + ns*(n). Then 
n 
S*(n) = .=1[ ]Z n js(j) 

Then 

n n n n 
rn=l 3k-=-I 3k-----1 j=l 

S*(n) - S*(n- 1) = ns*(n) = Z ds(d)" 

We shall show that S*(n) - S*(n- 1) is o(S*(n)). 
Fix e  (0, 1) and choose 

e(1 --e)" 

(5.3) M > 

(n-r)s(n-r) 
For any fixed integer r, 
choose N > ni 3 such that 

-- 1 as n tends to infinity. Hence we can 

Ins(n) - (n- r)s(n- r)l < ns(n) 

for 0 _< r _< M and n >_ N/M, and thus 

(5.4) (7/- r)(7/- r) > (1 - 

for 0 _< r _< M and n >_ N/M. 
For integers dl, d2 with 1 _< dl < d2 _< ni, and for n _> N, we have 

n n n(d2 - d) n ni 3 
d d dd - ni  M  

- M, 

7/ n 
and thus --22 < dl _M. 
Consequently, for n >_ N, if dl < .-. < dk are the divisors of n from the interval 
[1, _hi] we see that 

n n n n n n 
--- M < < M < .-. < d ni < dl 
(5.5) ni < dk -k dk- 
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Returning to the expression for S*(n) in (5.1), llOW assunling that n _> N, we 

have 

S*(,) 

by (5.3), (5.5) 

, () 
>- - Z ds(d) --I- Z //l(1-e)ê's by(5.4) 
dn/Jl 
1 1 
dln dln 
dKn/M d>n]M 
= (S*(n)-S*(n-l)) by (5.2). 
Thus 0  S*(n)- S*(n-1)  eS*(n), and so S*(n) 

6. COMBINING STAR W1TH EXPONENTIATION 
Theorem 6.1. Let T(x) = exp(S*(x)), where 
(a) S(x)(/ RTp withO<p< 1, 
(b) th« s(n) arc nonn«gativ«, and 
(c) liminf ns(n)p ' > 1. 
Then T(x)  RT and t(n- 1) 
t(n) 
Funh«rmore, if S(p)= oe, thon s(,) = o(t(n)). 
Pro@ S*(x)  RTp by Lemma 5.2; so T(x) 
s(n)  s*(n) we have lininf,s*(n)p  > 1. Thus t(n - 1)  t(n)p by Lemma 4.6. 
For the final assertion assume S(p) = . Then S*(p) = ; so from Lemma 4.4 
w have s*(n) = o(t(n)), and thus 
When p = 1 we can no longer assume S*(x) E RT just because S(x) E RTe. 
Instead we turn to xS*(x)/(1 - x) to final a well-behaved sequence of coecients. 
Theorem 6.2. Let T(x) = exp (S*(x)) where 
(.) s() e RT,, 
(b) the s(n) are nonnegative, and 
(c) ()v 1/. 
Thn ,S*(x)) e RT,. Fun«,o«, () = o(t()). 
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Pro@ Froln 
(6.1) s*(n) >_ s(n)  1/ 
it follows t.hat there exists a polynomial p(x) su«h that 
(6.2) S*(x) + p(x) + log(1-x) 
bas nonnegative coefiïcients. Then 
(6.3) R(x) = exp(S*(x) + p(x)) 
has nonnegative coefficients. Since (6.2) has nonnegative coefficients it also follows 
that the exponential of (6.2) has nonnegative coeftïcients, that is, 
(6.4) [xnl((1 - x)-ri(x)) _> o. 
Differentiating (6.3), and multiplying through by x, gives 
xrt'() = (s*'(x) + p'()). 
= (x(l- x)-(S"(x)+ p'(/)))- (1 - x)r(). 

We will use Lemma 3.3 with 
A(x) = 
B(x) = 
C(x) = 
For n larger than the degree of p(x), we have 
a(n) = [x'](x(1- 
x)-ls*(x) 
(6.5) = s*(1) +-.-+ ns*(n) 
From (6.1) we have S*'(1) = ec, and thus 
(6.6) p'(1) = o(  
From (6.5) and (6.6) it follows that 

x(1 - x)-(S*'(x) + p'(x)), 
(1-x)R(x), 
a'R'(x). 

+ x(1 - x)-'p'(x)) 

+ p'(1). 

(6.7) a(n)  s*(1) +---+ ns*(n). 
So by Lemma 5.3 and Proposition 2.4, A(x)  IRT:. This is condition (a) of Lemma 
3.3. 
B(x) bas nonnegative coelïicients by (6.4), and since R(x) also bas nonnegative 
coefficients, 
0 < b(n)= [x'q(1-x)R(x) <_ r(n)= [x']xR'(x)/n = c(n)/: 
so b(n) = o(c(n)). This gives condition (b) of Lemma 3.3. 
From (6.4) we also bave 
c(n) - c(n-1) = nf(n) - (- l)r(n-1) _> 0. 
Hence condition (c) of Lemma a.a holds. 
So, by Lemma a.a, c(.)  RTe, that is, a'R'(x)  RTe. Then R(x)  RT1 by 
Proposition 2.4. So from Corollary 3.2 we have 
T(x) = cxp( - p())- R(,)  ST,. 
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Finally, condition (c) implies S(1) = oc; so 
by Lemma a.a. Now 0 _< s() _< «() leads to 
[x]exp (S(x)) < []exp (S*()) : 
and thus s(n) : o(t(n)). 

[x]T(x) 

7. COMPTONS APPROACH TO LOGICAL LIMIT LAWS 

A class K: of finite relational structures is said to be adequate if it is closed 
under disjoint union and components, and, up to isomorphism, it has only finitely 
many structures of each size. Let p(n) count (up to isomorphism) the number 
of component structures in K: of size n, and let a(n) count the total number of 
structures in K: of size n. The combinatorial identity connecting the two counting 
fun¢tions a(n) and p(n) is 

(7.1) 
where 

A(x) = exp (P*(x)), 

A() = «)x , P'() = p'()x, 
p*(0) = 0, p*(n) = _,p(j)/k forn_> 1. 
jk=n 
The connection between adequate classes and (7.1) is very tight, for if p(n) is any 
nonnegative integer-valued function with p(0) = 0, then there is an adequate class 
K: with p(n) the count function for the components of , and the function a(n) 
satising (7.1) is the total count function for . 
Compton proved two main theorems for the purpose of finding cluses  with 
logical limit laws. We sume that  is an adequate class of relational structures 
with the counting functions a(n) and p(n)  described above. rthermore, we 
ume a(n)  0. 2 The striking feature of Compton's theorems is that he is able to 
prove logical limit laws just from knowing information about the counting function 
a(n). Note that if a(n)  RTp, then 0  p  1 since the a(n) are integers. We 
only consider 0 < p  1 since Compton's method does not work for the ce p  0 
(The ce p = 0 requires more knowledge about  than just the count functions to 
determine if there is a logical limit law.) 

Theorem 7.1 (Compton, 1987/1989). If a(n) E RT1, then ] bas a monadic 
second-ortier zero-one law. 

Theorem 7.2 (Compton, 1989). If a(n)  RTp, where 0 < p < 1, and if there 
exist K and C such that 

(7.2) a(n- k) <_ Cp k for K _ k <_ n, 

then ] has a monadic second-ortier limit law 

2This is the saine as requiring that the gcd of the sizes of the components be 1 
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8. A USEFUL COROLLARY 

For applications of Theorem 7.2 we use thc following. 

Corollary 8.1. Suppose a(n) Ç RT v and a(n- 1) 
second-order limit law. 

Then IC bas a monadic 

Proof. Choose K > 1 such that 

n>h 

Now suppose K _< k _< n. 
Case 1: If n - k > K, then 

{ a(n) > 0, and 
a(,t- 1) < 
a(,) 

(- k) 

Case 2: If n - k < K, then 

a(- k) 

.(n - k) .(K) 
(/) a(-) 
< a(n- k) 
- (iç) 

by Case 1 

Thus Theorem 7.2 applies. 

9. APPLICATIONS TO LOGICAL LIMIT LAWS 

Now we use out results to improve the scope of the conditions on p(n) that lead 
to classes of structures covered by Cmpton's theorems. 

Theorem 9.1. If p(n) E RT1, then IC bas a monadic second-order zero-one law. 

Proof. This follows flore Theorems 6.2 and 7.1 after noting that p(n) is always a 
nonnegative integer. [] 

This theorem yields many classes of structures that, before, were not known to 
have a 0-1 law. For example, if 

with d < 1 and a, c > O, then one has a monadic second-order 0-1 law. Previously 
the known comprehensive general collection of count functions p(n) that implied 
such a law was the family of polynomially bounded p(n), that is, p(n) = O(n c) 
for some c (see Bell [1]). Now we have entire families of superpolynomial count 
funct.ions p(n) that guarantee such laws. 
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Example 9.2. Let/ be the class of finite directed graphs G = (G, R) satisfying 
the first-order conditions 
Vx(3y(yn)  xP), 
wvw((n)(n)(z)  ()), 
væ((xæ)  (x)). 
Such a digraph consists of an equivMence relation plus a possibly empty collec- 
tion of vertices that each have but one edge attached to them, md that edge is 
outgoing to an element in the equivalence relation. The vertices comprising the 
equivMence relation are precisely those in the range of the biny relation R. The 
indecomposable members of K e thosc with a single equivence class. It is easy 
to see that p(n) is the number of partitions of the integer n, which we know from 
the fmus results of Hardy and Ramujan to satisfy 
1 _len/" 
p()  
By Threm 9.1 we see that  has a mouadic second-order 0-1 law. 
Theorem 9.3. Suppose 0 < p < 1. If 
() p()   d 
(b) imp()p  > , 
then  has a monadic second-order limit law. 

a(n - 1) 

Proof. From Theorem 6.1 we know a(n) E RTp as well as a(n) -< p; so Corollary 
8.1 applies. [] 
Prior to Theorem 9.3 ail applications of Compton's Theorem 7.2 were based on 
the asymptotics of Knopfmacher, Knopfmacher, and Warlimont [11] that say if 
(9.1) p(n) = C1/3 n ÷ O('yn), 0 ( "y ( /3, C1 > O, 

then 

a(n)  C2/3e2V/n 3la, for some C2 > 0. 
These results were proved by the well-known use of Cauchy's integral theorem for 
asymptotics. In this case, 
a(n)- 1 c dz 
2ri exp (P*(z))  z+l . 
From the hypothesis (9.1) we immediately have p = 1//3 and p(n) E RTp as 
well as liminfnp(n)p  > 1. Thus Theorem 9.3 covers condition (9.1). But we 
?2--+Oe 
can say much more. In the following corollary, we focus on generalizing the C/3  
asymptotics and drop any reference to an error terre. 
Corollary 9.4. For # < 1 </3 and C > O, or # = 1 </3 and C > 1, the condition 
p(n) ~ C/3/,  
guarantees that  has a monadic second-order limit law. 
Proof. From Proposition 2.4 we see that p(n)  Tp, where p = 1//3, and the 
condition lira inf np(n)p  > 1 is readily verified. Thus Theorem 9.3 applies. [] 
?2--+Oe 
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Example 9.5. For a fixed positive integer k, let ]C be the class of finite k-colored 
linear forests. (The componcnts of linear forests are chains. If k = 4, then one can 
think of a member of ]C as a collection of DNA fragments.) Then 
(n) ~ 
so by Corollary 9.4 we know that ]C has a monadic second-order limit law. 
Example 9.6. For h a fixed positive integer, let ]C be the class of finite forests of 
planted plane trees of height at most h, that is, the class of finite graphs whose 
components are planted plane trees of height at most h. 
The basic study of thc generating flmction P(x) := OEp()x ' for finite planted 
plane trees can be found in thc 1972 paper [9] of de Bruijn, Knuth and Rice. From 
thcir work we know that 
f(x) 
P(x)- g(x) 
where f(x) and g(x) are the polynomials given b,v 
(1 + -:qT) h - (1 - v:-) h 
f(x) = 2x. xii- 4.r ' 
(1 ÷ 1V/-ï-- ) h+l- (1 - V/ï---) h+l 
o(x) = vq-4« 
They also note that the degree of g(x) is d = [h/2], and that g(x) has d distinct 
positive roots ri given by 
ri =  sec 2 ï for 1 _ j _< d. 
Of course we need h >_ 2 in order for g(x) to have any roots. Clearly 0 < ri < .-. < 
rd, and for h _> 3 the smallest root ri lies in the interval (1/4, 1/2]. 
Given that h _> 3, for some c =fi 0 we have 
g(27) - C(27-- rl)---(27--Td). 
So by the method of partial fractions, 
d 
g(x)l =  x-cJ ri where Cj = C -1 H(rj -- ri) -1. 
j=l 
From this we easily have 
p(n) = Ix n]f(x) 
d 

= -cs(,»)/,-? + 

h-i 
forn> [--] (=deg(f)) 

f(l'l) (Ln ' 
" --C1 -- 
/'1 \/'1 / 
and thus for h >_ 3 the class ]C has a monadic second-order limit law by Corollary 
9.4. In the cases h = 1, 2 the values p(n) are uniformly bounded, and thus one has 
a monadic second-order 0-1 law. 
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Rernark 9.7. If we change the condition/z < 1 in Corollary 9.4 to/z > 1, then we 
tan find exarnples of classes K with such a count function p(n) that fail to have a 
first-order limit law. 
Rernark 9.8. We tan construct, for any given p with 0 < p _< 1, an infinite sequence 
/Çm of classes of finite relational structures with monadic second-order limit laws 
such that the count functions pro(n) and a(n) for /Ç are in RTp and growing 
infinitely faster at each successive step; that is, we have 
Pm(') = o(am(n)) alld am(,t)=O(pm+l(n)). 
To construct the sequence we start with 
p0(n) = Il/phi, for n _> 1. 
Then 
a0(n) = [zn]exp(Pô(z)) 
gives p0(n) = o(a0(n)) by Theorem 6.1 or 6.2. Now observe that by modifying 
a0(n), by simply setting a0(0) to 0. we have a sequence that tan be used as a p(n), 
satisfying the premises of Theorenl 6.1 or 6.2. lnductivelv define Pk+l and ak+l by: 
p+l (0) = 0 and 
pe+l(,,) = [x]exp(A.(x)). for u_> 1, 
a+l(n) = [z']exp(Pî+l(Z)), for n >_ 0, 
where P(x) = -pk(n)z  and A(z) = -, ak(n)z'L Now take classes Km with 
counting functions ara(n) and pro(n). 
hl the case that p = 1 these classes have, for ru >_ 1, both counting functions 
with superpolynomial growth, growing far faster than the classes with polynomially 
bounded p(n) that were, with minor exceptions, the only ones that were previously 
known to have a inonadic second-order 13-1 law. Likewise, with 0 < p < 1. the 
classes/Çm, for ru _> 1, grow far faster than any examples that we knew before with 
a monadic second-order limit law. 
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COMBINATORICS OF ROOTED TREES AND HOPF ALGEBRAS 

MICHAEL E. HOFFMAN 

ABSTRACT. We begin by considering the graded vector space with a basis 
consisting of rooted trees, with grading given by the count of non-root vertices. 
We define two linear operators on this vector space, the growth and pruning 
operators, which respectively raise and lower grading; their commutator is 
the operator that multiplies a rooted tree by its number of vertices, and each 
operator naturally associates a multiplicity to each pair of rooted trees. By 
using symmetry groups of trees we define an inner product with respect to 
which the growth and pruning operators are adjoint, and obtain several results 
about the associated multiplicities. 
Now the symmetric algebra on the vector space of rooted trees (after a 
degree shift) can be endowed with a coproduct to make a Hopf algebra; this 
was defined by Kreimer in connection with renormalization. We extend the 
growth and pruning operators, as well as the inner product mentioned above, 
to Kreimer's Hopf algebra. On the other hand, the vector space of rooted 
trees itself can be given a noncommutative multiplication: with an appropri- 
ate coproduct, this leads to the Hopf algebra of Grossman and Larson. We 
show that the inner product on rooted trees leads to an isomorphism of the 
Grossman-Larson Hopf algebra with the graded dual of Kreimer's Hopf alge- 
bra, correcting an earlier result of Panaite. 

1. INTRODUCTION 

In recent work on renormalization of quantum field theory, D. Kreimer and his 
collaborators [1], [3], [13], [14], [15], [16] introduce a Hopf algebra (here denoted 
K) whose generators are rooted trees. Various other Hopf algebras based on 
rooted trees have appeared in the literature, in particular that of R. Grossman 
and R. G. Larson [10], which F. Panaite [18] connected to K- The proof of the 
principal result of [18] actually contains an error due to the confusion of two kinds 
of multiplicities associated to triples of rooted trees. In this paper we show how 
to correct Panaite's result, while clarifying the combinatorial significance of these 
multiplicities. 
Kreimer's Hopf algebra  admits a derivation called the growth operator, 
which is important in describing the relation of this algebra to another Hopf algebra 
studied earlier by A. Connes and H. Moscovici [4]. We introduce a complementary 
derivation called the pruning operator. In fact, we find it easiest to start (in §2) in 

Received by the editors June 25, 2002 and, in revised form, February 24, 2003. 
2000 Mathematics Subject Classification. Primary 05C05, 16W30; Secondary 81T15. 
Key words and phrases. Rooted tree, Hopf algebra, differential poset. 
The author was partially supported by a grant from the Naval Academy Research Council. 
Some of the results of this paper were presented to an AMS Special Session on Combinatorial 
Hopf Algebras on May 4, 2002. 

@2003 American Mathematical Society 

3795 



3796 MIÇHAEL E. HOFFMAN 

the vector space k{9"} of rooted trees rather than in n-. There we have growth 
and pruifing operators (denoted by .ql and  respectively), and for each pair of 
rooted trees t, t' with Iii _< It'l (where ]t] is the number of vertices of t) there are 
natural multiplicities n.(t;t ) and -m(t;t ) associated with  and 3 respectively. 
A comparisou of these multiplicities using symmetry groups of rooted trees leads 
to t.he definitiou of an immr product with respect to which  and 3 are adjoint. 
The operators  and  are very similar t.o the adjoint operators that appear in 
Stanley's theory of differential posets [21], [22], and tan be described in terres of 
Fomin's somewhat nlore g(nel-al theory of d/lal graded graphs [6], [7], [8]. The 
techniques of Stanley and Forain are used to obtain various results about n and m. 
In §3 we extend the growth and pruning operators (as well as the inner product) 
to the Hopf algebra 9-CK. In this setting the growth and pruning operators are not 
quite adjoint, but the deviation ff'oto adjointness is easily described (Proposition 
a.a). w' also describe the duals of the growth aud pruning operators. In §4 
we extend the inultiplicities n and m to nmltiplicities n(t, t2; t3) and m(tt, t2:/3) 
associated to triples of rooted trees t, t2, fa with Itl + If2[ = It3[. We then give an 
explicit isolnorphism of the Grossuian-Larson Hopf algebra onto the graded dual 
of IKn- (Proposition 4.4), providing a corrected version of Panaite's result. We also 
show how the isomorphisln gives another description of the duals of the growth and 
pruning operators. 
Iu addition to thc references cited above, two recent articles treat aspects of 
IKK hot covered here: sec [2] for connections betweeu Kreimer's Hopf algebra and 
earlier work on 1Runge-Kutta methods, and [5] for an analvsis of the prinfitives of 
IKh-. The author thanks the referee for bringing them to his attention. 

'2. THE VECTOR SPACE OF ROOTED TREES 

A rooted tree is a partially ordered set (whose elemeuts are called vertices) with 
a unique greatest element (the root vertex), such that, for any vertex v, the vertices 
exceeding v in the partial order form a chain. If v exceeds w in the partial order. 
we call w a descendant of v and v an ancestor of w. If v covers w in the partial 
order (i.e., v is an ancestor of w and there are uo vertices between v and w in the 
order), we call w a child of v and v the pareut of w. 
We can visualize a rootcd tree as a directed graph by puttiug an edge fl'om each 
vertex to each of its children: the root is the only vertex with uo incoming edge. We 
call a vertex terminal if it has no outgoing edges (i.e., uo children). The condition 
that the set of ancestors of any vertex forms a chain insures that this graph has no 
cycles. For a finite rooted tree t, we denote by It] the number of vert.ices of t: let 
T be the set of finite rooted trees, and T, = {t  T : Itl = n + 1}. For example, 
T0 = {*}, where  is the tree consisting of only the root vertex, and below are the 
four elements of T3, with the root placed at the top: 

We can define a partial order -< on the set ç itself by setting t _ t' if t can be 
obtained from t' by removing some non-foot vertices and edges; of course t  t' 
implies [t I 5 If'l. Evidently ' covers t for this order exactlv in the case when t can 
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be obtained by removing from t' a single terminal vertex and the edge into it. In 
this case we write t <] t'. 
If t <l t , we can get from t  to t by removing a (terminal) edge, and from t to t' by 
adding an edge (raid accompanying terminal vertex). This leads to the definitions 
of two numbers associated with the pair (t, t): 
.n(t; t') = I{vertices of t to which a new edge can be added to get 

and 

re(t; t') = [{edges of t' which whell relnoved leave 
That these two numbers are not always equal can be seen from the example of 

where ,n(t; t') = 1 and re(t; t') = 2. 
The relation between the numbers ri(t; t ) and re(t: t ) can be clarified by intro- 
ducing symmetry groups of trees. For a rooted tree t, let V(t) be its set of vertices: 
then for each v E V(t), there is a rooted tree tv consisting of v and its descendants 
with the order inherited from t. We call this the subtree of t with v as root. For 
v  V(t), let SG(t, v) be the group of permutations of identical branches out of v, 
i.e., if {v, v2,..., vk} are the children of v, then SG(t, v) is the group generated 
by the permutations that exchange tv, with tj when they are isomorphic rooted 
trees. The symmetry group of t is the direct product 
sa(t)= I71 sc(t,v). 
vV(t) 
For a given v  V(t), let Fix(v, t) <_ SG(t) be the subgroup of SG(t) that fixes v; 
note that Fix(w, t) <_ Fix(v, t) whenever w is a descendant of v. 
Now suppose t <1 t , and let v  V(t) be such that, when a new edge and terminal 
vertex w are added to t at v, the result is t . If Orb(v, t) is the orbit of v under 
SG(t), then evidently 
ISC(t)l 
,ri(t; t')= I Orb(v,t)[--ISC(t)/Fix(,t)[- 
On the other hand, if Orb(iv, t') is the orbit of iv  V(t') under SG(t'), then 
Isc(t')l 
m(t;t') =/Orb(iv, t')l = ISÇ(t')/Fix(iv, t')l = I Fix(iv, t')l" 
But there is an evident identification of Fix(v, t) with Fix(iv, t'); so we have the 
following result. 
Proposition 2.1. If t <1 t', the'n ISG(t)lm(t; t') = ri(t; t')lSG(t')l. 
Let 

,_>o 
be the graded vector space (over a field k of characteristic O) with basis consisting of 
rooted trees: we put the rooted tree t in grade Itl - 1. We define two linear operators 
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on k{'J'} as follows. For n _> 0, the growth operator 9l- k{'J', } --, k{'J',+ } is defined 
by 
(1) 91(t) = 
t <t  
and for n >_ 1 the pruning operator - k{,} - k{,_l } is given by 
(2) (t) = 
we set (.) = 0. Then  and 9l satisfy the following commutation relation. 
Proposition 2.2. As operators on k{9"}, [,91] = , where  is the operator 
given by (t) = t]t. 
Proo]. It suffices to show (t) - (t) = ]tt for any rooted tree t. Let V(t) = 
{v,..., vn, v+,..., Vltl} be the vertices of t, with v terminal for 1 < i < n. Then 
Itl 
where t is the tree obtained from t by adding a new edge and terminal vertex to t 
at v, and t (0 cornes from t by relnoving the edge that ends in v. Then (t) is 
(t)= t+  (t) (') =«t+ E (t)(')' 
=1 i=1 ljn,ji i=1 ljn,ji 

while 913(t) is 

91(t(J) --   (t(3)). 
j=l =1 lj(_n,jyti 
Since (t) (j) = (t(J)) for i  j, the conclusion follows. [] 
Now we can endow k{9"} with an inner product by setting 
(t,t') = 
for any rooted trees t, t p. 
Proposition 2.3. The operators 91 and  are adjoint with respect to the inner 
product (., .). 
Proof. It suiîïices to show 
(91(t),t') = (t, (t')) 
when t < t p (otherwise both sides are zero). In this case, we have 
(91(t),t') = n(t; t')(t', t') = 
from equation (1) and 
(t, (t')) = re(t; t')(t, t) 
from equation (2); but then the result follows by Proposition 2.1. [] 
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Putting the last two results together gives the following. 
Proposition 2.4. For rooted trees il and t2, 
{0, iftl  t2, 
(.q2(tl),.q2(t2)) -- ((tl),(t2)) ---- ItlISG(t)[, iftl = t2 = t. 
Proof. This follows from the calculation 
(.¢(tl), .¢(t2)) --((tl), (t2)) = (il, (.¢- .¢)(t2)) ----(tl,V(t2)) ----It21(tl,t2). 

Remark. The second alternative of this result can be written 

(t;t')21SG(t')l-  m(t";t)U'SG(t")l = 
t <3t  t"<3t 

or, dividing by Isc(t)l, 

 n(t;t')m(t:t') - _, n(t";t)m(t";t) = Itl 
t <3t  t"<3t 

for any rooted tree t. 

(3) .(t)=  (t;t')t' 

and 

(4) ç(t') =  ,,(t;t')t. 
It'l=ltl+k 

With these definitions, we have the following result. 

Proposition 2.5. Let t, t' be rooted trees with Itl < If'l. 
1. n(t;t')lSG(t')l = ISG(t)lm(t;t'). 
2. If Itl < k < It'l, then 

Then 

u(t;t')=  n( t; t")n( t"; t'), 

and similarly for n replaced by m. 
3. n(t;t') and m(t;t') are nonzero if and only ift _ t'. 

Proof. The first part follows immediately from equations (3) and (4): 

ri(t; t')lSG(t')l = (t', fftlCl-lt'(t)) = (q3't"-'t'(t'), t) = re(t; t')lSG(t)l. 

We Call extend the definitions of re(t; t') and ri(t; t') to any pair of rooted trees 
t, t' with It'l - Itl = k >_ 0 by setting 
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For the second part. we have for [ri 5 k < 
ri(t; t')= 
(-'(t), 

=Z 

=Z 

( 91-Il ( t ), re(t"; t')t") 

(91»-Itt (t), t") Isa(t")l , 
Isa(t")l  *(;t') 

n(t:t"),,(t";t'). 

(For the corresponding equation with m replacing n, reverse the roles of 9l and 3 ) 
FinMly, the third part is evidcnt for It'l -Itl = 1 and can be proved by induction 
o,1 It'l- Itl using the seco,,d part. [] 
R«mark. Thc second and third t)arts say that 9" is a weighted-relation poset, in 
the terlninology of [11], for either of the weights ri(t: t') or re(t: t'). In fact, 9" with 
weights n(t; t') is discussed as Example 7 in [111. In the terminology of [7], 9" with 
multiplicities ri(t; t') and 9" with multiplicities re(t; t') are a pair of graded graphs 
that are r-dual for the sequence r = (0, 1, 2,... ). 
If t  t', we can think of ri(t; t') as counting the ways of building up t' from t by 
adding new edges and ternlinal vertices, and re(t; t') as counting ways of getting 
from t' to t by removing terminal edges. In particular, since   t for every rooted 
tree t, we can think of n(-:t) as the number of ways to build up t, and m(.:t) 
as the number of ways to tear it down. A more precise formulation can be given 
using the idea of labellings of trees: a labelling of a rooted tree t is a bijection 
f " V(t) ---, {0, 1,...,Itl} such that f(v) > f(w) whenever v is a descendant of w 
(necessarily f sends the root vertex to 0). We call labellings f and g equivalent if 
le = g for SOllle ¢ Ç SG(t). 
Proposition 2.6. Let t be a rooted tree. Then t bas m(: t) labellings and n(: t) 
labellings mod equivalence. 
Proof. First note that ]SG(t)I(: t) = ml:t) by the first part of Proposition 2.5 
since  has trivial symmetry group. It follows from the discussion of [11, Ex. 7] that 
n(; t) counts labellings mod equivalence, and the statement about m(; t) follows 
since each equivalence class of labellings has ISG(t)I elements. [] 
Remark. The "Connes-Moscovici weight" [1], [15] or "tree multiplicity" [2] of t is 
n(:t). Cf. [20, Sect. 22] and [12, Ex. 5.1.4-20], where a hook-length formula for 
the number of labellings of t is given: this is m(: t). 
In [22] Stanley defined the notion of a sequentially differential poser (generalizing 
his definition of a differential poser in [21]). A sequentially differential poser P is 
a locally finite graded poser with a single element Ô in grade 0, so that the linear 
operators 
U(p) = Z p' ana D(p) = Z p' 
pp' p' p 
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on k{P} satisfy the identity (DU - UD)(p) = rjp for any p E P of tank j: here 
r0, ri,.., are nonnegative integers. The results of [22] can be applied to 9" (with 
ri = j + 1), provided we replace U and D with 9l and 3 respectively, and suitably 
reinterpret the statements of theorems to incorporate multiplicities. For example, 
for x E P Stanley writes e(x) for the nurnber of saturated chains from Ô to x, but 
in the proofs e(x) really appears as the inner product of x with ukÔ, where k is the 
rank of x: so for a rooted tree x  7k we replace e(x) by 
(91o,:) = ,(o; :)(:, :) = ,(o; :)lSG(:)l = 
Let w = WlW2...w be a word in 9l and 3, and let x  7k. Clearly (w.,x) = 0 
unless (a) for each 1 <_ i _< r, the nunlber of 3's in wWi+l ...w does not exceed 
the nunlber of 91's; and (b) the number of 91's minus the number of 3's in w is k. 
In this case we call w a valid x-word, and we have the following result. 
Proposition 2.7. Let x  7k, w = wv.. w a valid x-word. Let S = {i  wi = 3}. 
For each i  S, let ai = I{J " J >_ i, wj = },, bi = I{J  J > i, wj = 91},, and 
ci = bi - ai. Then 
(w"x)=m(';x) H (ci + 2) "2 
Pro@ Replace U, D by 91,  in Theoreln 2.3 of [22]. [] 
This result has the following corollary (cf. Theorem 1.5.2 of [7]). 
Proposition 2.8. For any rooted tree x  9"k and nonnegative integer a, 
a+l 
E m(x;t)n(.:t) = n(.:x) H (k 2 + i . 
i=2 
Pro@ In Proposition 2.7 set w = 3«91 «+k to get 
(3a91+k"x) = m(':X)a (k + i) "2 
i2 

Now the left-hand side can be expanded as 
(91a+k"91«(X)) = E n(x;t)(91«+"t) = 
Itl=k+«+l 

where we have the first part of Proposition 2.5 in the last step. Hence 
a÷l ) 
_, m(æ;t)lSCæ)ln(o:t) = m(o;æ) 1-I (k +i 
2 " 
i=2 
Itl=k+a+l 
and dividing by [SG(x)[ gives the conclusion. 
Remark. In the case x =., this result becomes 

Itl=«+l 
cf. [7, Cor. 1.5.4]. 

m(,;t)n(.;t) = 

, (o,t)lsct)l=- ()  
Itl=«+l i=2 

[] 
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In [11, Ex. 7] it is shown that ltl=k+l n(.: t) = k!. Further SUlll formulas involv- 
ing rt(.:t) appear in [15, Sect. 5] alld [2, Sect. 5]. A result of [22] gives a formula 
for -dtl=k+l m(.: t). To state it we will need some definitions. Let Inv(k) be the 
set of involutions in the group E of perlnutations of {1.2 .... , k}. For «  E, call 
i a weak excedance of « if «(i)  i; let Wex(«) be the set of weak exceda.nces of 
«. For «  E and i  {1,... ,k}, let («,i) be the number of integers j such that 
j < i and «(j) < «(i). Then we have the following result. 
Proposition 2.9. With the definitions above, 
It[=k+l aInv(k) iWex(a) 
Pro@ In the proof of Theorem 2.1 of [22], replace U, D with , : in the oenclu- 
sion, this replaces ra.,kz= 
For exalnple, a sure over the four invohltions 123, 213, 132, and 321 of Ea gives 
.:t)=l-2-3+l.3+l-2+l-1 =12. 

3. KREIMER'S HOPF ALGEBRA 

In this section we discuss the Hopf algebra 9Q,- defined by D. Kreimer and his 
collaborators [ll, [31, [13], [141, [15], [16] in connection with renormalization. As an 
algebra, 9Q( is generated by rooted trees; so as a vector space, 9Q,- is generated 
by monomials in rooted trees, i.e.,"forests" of rooted trees. For a rooted tree t, we 
give the corresponding generator degree ]t[ in 9Q(: in degree 0, 9Q( is generated by 
the unit element 1. For example, the degree-3 part of 9K is generated as a vector 
space by the four elements 

ooo.  I . t, and A 
There is a linear map B+  9Q( -- k{7} which takes a forest to a single tree with 
a new root vertex connected to ail the roots of the forest: e.g., 

The map B+ takes the degree-n part of 9K onto k{n}: if we set B+(1) = .. 
then B+ is a vector space isomorphism. We write B_ for the inverse of B+. On 
the other hand, except for the degree shift,  is just the symmetric algebra on 
k{7}. Thus, ifT = dimk{7} = I1, w hv 
1 
(5)  Tx =  (1 - x) T- 
n0 nl 
from which we can compute recursively T0 = 1, T = 1, T = 2, Ta = 4, T4 = 9, 
etc. (see [19] for more information). 
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To define the bialgebra structure on 9£K, we let the counit send ail elements of 
positive degree to 0, and the unit element 1 in degree 0 to 1 E k. The comultipli- 
cation A bas A(1) = 1 ® 1, 
(6) A(t) = t ® 1 + (idcB+)A(B_(t)) 
for a rooted tree t, and /k(tlt2 ""tr) = A(tl)/k(t2)'"/k(tn) for monomials 
tir2  " " tn. 
Equation (6) givês a rêcursive definition of thê coproduct, but therê is also a 
nonrêcursivê dêfinition in terres of cuts. A cut of a rootêd trêê t is a set of edges of 
t. A cut is êlelnentary if its cardinality is 1. When thê elements of a cut C of t are 
removêd, what remains is a collection of rooted treês: thê onê containing the foot 
is dênotêd Re(t), and the remaining rootêd treês form a monomial denotêd pe(t). 
For example, if t is thê tree 

;-.; 
and C consists of thê dotted edgês, theu 

RC(t) = t aud 

pC(t) = ,  

Thê order of a cut C of t is the largest numbêr of edgês in C between thê root 
of t and any of its terminal vêrtices: a cut of order at most 1 is callêd admissible. 
Thê êmpty cut  is thê only cut of ordêr 0 (note that R°(t) = t and P°(t) = 1). 
Thê following formula for A(t) is provêd in [3]. 
Proposition 3.1. For a rooted tree t, A(t) can be written 
A(t)=t®l+ ï pC(t)®RC(t). 
C admissible cut of 
Wê can dêfinê growth and pruning opêrators N and P on 9£; as follows. Thê 
growth operator N is simply 92 extendêd as a dêrivation, i.e., 
N(t,t2...t)= -t,...92(ti)...t. 
i=l 
We also dêfinê P as a dêrivation, but set P(t) = q3(t) only for Itl > 2; we put 
P(.) = 1. If D : 9£; -- 9£; is thê extension of  as a dêrivation (i.e., thê linêar 
map that multiplies a monomial by its degrêe), thên thê identity 
(7) [P, NI = D 
holds. To provê equation (7), note that both sides are derivations, and so it suïcês 
to prove it for rootêd trêês t; but in that case, (7) follows from Proposition 2.2. 
The map B+ intêracts with the growth and pruning operators as follows. 
Proposition 3.2. For monomials u of 
1. B+P(u) = q3B+(u), 
2. B+N(u) = 92B+(u) - B+(.u). 
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Proof. Suppose u = tl .-. tk with each Itil >_ 1. Then applying B+ to 
P(u) = P(tl)t2...tk +tiP(t2)t3...tk +...+ti...tk_iP(tk) 
gives a sure of rooted trees that includes all those obtained by removing terminal 
edges of B+(u), and the cases with ri. = * (hence P(ti) = 1) work correctly: these 
are exactly those cases where an edge coming out of the foot of B+(u) is terminal. 
If u = 1, then B+(1) = (.) = 0 = B+P(1). So in any case, B+P(u) coincides 
with B+ (u). 
Now for u = ri ---tk, B+ applied to 
N(u) = N(tl)t2...tk + tlN(t2)t3...t +...+ tl ...tk_lN(t) 
will include all those trees obtained by adding a new edge to each vertex of B+ (u) 
exccpt one- the "new" root vertex. Thus, B+N(u) is missing the terre obtained 
by adding a new edge to the foot of B+(u), namely B+(..u). On the other hand, if 
u = 1 we have B+N(1) = 0 = 9l(.) - B+(.). [] 
We can extend the inner product of the previous section to 9(K by setting 
(u,u2) = (B+(ul),B+(u2)) 
for monomials Ul, u2; thcre is no ambiguity since (B+(t), B+(t')) = (t, t') for rooted 
trees t, t'. With this dcfinition, we can state the adjointness relation between P and 
N. 

Proposition 3.3. On 9K, the adjoint of P with respect to the inner product above 
is N + 2il.. where 2il. is the operator that sends u to .u; equivalently, the adjoint 
ofN isP ô 
Proof. Let al, u2 be monomials of :Kg. Then 
(ul, P(u2) ) = ( B+(ul ), B+P(u2) ) = ( B+(ul ), 3 B+ (u2) ) = ( 9B+ (ul ), B+ (u2) ), 
from which the first statement follows using the second part of Proposition 3.2. For 
the second statement, note that  is adjoint to 211o. [] 
We now compute the characteristic polynomial of the restriction PNk of PN to 
the degree-k part of 9Q. Let C(L, A) = det(AI - L) for a linear transformation 
L. 

Proposition 3.4. For k >_ 1, 

Ch(PN«,A)=(A-( k 

I r ) 
q- 1)) Îl A - -(k - j) , 
2 r=0 j =o 

where as above T is the dimension of the degree-i part of 
Proof. We follow the proof of [21, Theorem 4.1]. Evidently PN1 is the identity, 
and so the result holds for k = 1; assume it inductively for k _> 1. From elcmentary 
linear algebra, 
Ch(NPk+I,A) = )T+--T Ch(PNk, A), 
while from equation (7) we have 
C(PNk+I, A) = C(NPk+I, A - (k n u 1)) 
since D+ = (k + 1)I. The induction step then follows. [] 
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The preceding result implies that Nk is injective for all k > 1 and that Pk is 
surjective for k _> 2; of course P1 is also surjective. In addition, the maximal 
k-I-1 
eigenvalue of PN is ( 2 )" In fact, the element 

f = N-(')= E n(.t)t 

is a corresponding eigenvector. To see this, note that 

PNk(fk) = P(fk+l)=  t'  

n(.:t)m(t';t) 

= E n(':t')(k+l) t'--- (k-t-l) 
2 2 
I«1--¢ 

where we have used Proposition 2.8. The f are the "naturally grown forests" of 
[3] (where they are denoted 5). 
The following result, which describes how N behaves with respect to the coprod- 
uct, is essentially [3, Prop. 6]. We give the proof since it can be stated concisely 
and illustrates the use of Proposition 3.1. 

Proposition 3.5. AN = (N ® id + id ®N + Mo ® D)A. 

Pro@ Since both sides are derivations, it suflïces to show that 

AN(t) = (N ® id + id ®N + Mo ® D)A(t) 

for any rooted tree t. As in the proof of Proposition 2.2, write N(t) = Y-i ti, where 
each ti is the result of adding an edge to t. Then 

AN(t) = Eti ® 1 +  
= N(t) ® 1 + -. 

E PC'(ti) ® Rc(ti) 
Ci admissible cut of t, 
E PC'(ti) ® 

z C admissible cut of ti 
Now each eut Ci of ri either includes the "new" edge or it does not. Suppose first 
that Ci does not include the new edge. Then Ci corresponds to a eut C of t and 
either PC'(ti) ® RC(ti) is a term in pC(t) ® NRC(t) (if the new edge is in the 
component of the foot) or a term in NpC(t) ® Re(t) (if it is not). Together with 
the leading term N(t) ® 1, these give all the terms of (N ® id + id ®N)A(t). 
Now suppose that Ci includes the new edge of ti. If C is the cut of t given by 
Ci minus the new edge, then the new edge must have been attached to a vertex of 
Re(t) (by the definition of admissibility), and so 
PC'(ti) ® RC'(ti) = .Pe(t) ® Re(t). 
Since (for each admissible cut C of t) there are IRC(t)l vertices to which the new 
edge could be attached, terms of this form contribute (Mo ® D)A(t). [] 

Remark. It follows from this result that the f, k > 1, generate a sub-Hopf-algebra 
of :KK. This Hopf algebra is isomorphic to the graded dual of the universal en- 
veloping algebra of A , the Lie algebra of formal vector fields on R that vanish to 
order 2 at the origin (see [3]). 
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Since 9lu is a locally finite commutative Hopf algebra, its graded dual 9fï( r is a 
locally finite cocomnmtative Hopf algebra, hence (by the results of [17]) the univer- 
gr gr 
sal enveloping algebra of the Lie algebra (K K ), the prinfitives of K K . Primitives 
of Kï( r are dual to indecomposables of KK, and so are linear combinations of ele- 
mcnts Zt for rooted trees t, where (Zt, u> = 5t, for monomials u E KK. The duals 
of N and P can be described as follows. 

Proposition 3.6. 
(s) 

for Itl >_ 2, and 
(9) 
gr 
for w, v  9-f tv . 

1. N* is given by N* ( Zo ) = O. 
N*(Zd= y .(t':t)Z,, 

W 
N*(,) = (N*) + (N*) + 
o 

2. P* (w) = Z.,L, for w e 96 . 
Pro@ To prove the statements about N*(Zt), note that (N*(Zt),u) = (Zt,N(u)) 
is zcro unless u is a scalar multiple of t', for some t' <1 t; but. then equation (8) 
follows from equation (1). Equation (9) follows from Proposition 3.5 since the 
multiplication in 9f  is induced by A. 
For the second part, let t be a rooted tree. If we write P(t) = -i t(') as in the 
proof of Proposition 2.2, then evidently 
 ® (t) = . ® 
i 
are (by Proposition 3.1) exactly those terres of A(t) of the form  ® t'. Now let 
u = tt2 ..- t, be a monomial of 9rg. Then 

and thus 

/() = ]-I/(t) 

= 1-I(1 ®ri + ® P(t)+ ... ) 
z---1 
= 1 ®t...t, + ® (P(t)t...t, +... + t...t,_P(t,)) +... 
= 1 ®u+  P(u) +... 

(Z.w,) = (z. ® ,/x()) = (, p()) = 
for all w G 9fï r and monomials u of 9rg. 
gr 
Remark. The Lie algebra P(gfg) is in fact free: see [5]. 

4. THE GROSSMAN-LARSON HOPF ALGEBRA 

We can define a noncommutative multiplication on the graded vector space k{9"} 
as follows. Let t, t' be rooted trees, and suppose B_(t) = tt2.., t.. There are Itï 
rooted trees obtainable by attaching each of the k rooted trees t, t2 ..... tk to some 
vert.ex of t' (by a new edge): let rot'  k{9"} be the sure of these trees (if t = . we 
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define t o t' to be t'). For example, 
foI=    
while 
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This product lnakes k{9" / a graded algebra: note that for t E k{9"n / and t' E k{9"m ), 
we have t o t'  /,'{9",+, }. The element   9"o is a two-sided identitv. Note also 
that 

for any rooted tree t. 
Now define a coproduct A: k{9"}  k{9"} ® k{9"} by 

(10) 

a(t) =  B+(tl) ® 

where B_ (t) = tl---tk and the Sllln is over pairs (I, J) of (possibly elnpty) subsets 
I, Jof {1 .... ,k} such that IUJ = {1 ..... k}: ri meansthe product oft for 
i G I. The following result is proved in [10] and [9]: the main things to check are 
the associativity of the product o [10, Lelnma 2.6] and the compatibility of the 
coproduct with o [10, Lenmm 2.8]. 

Proposition 4.1. The vector space k{9"} with product o and coproduct A is a 
graded Hopf algebra 9fOL. 

Since the coproduct A is cocommutative, by results of [17] it follows that 9£C;L 
is the universal enveloping algebra on its Lie algebra [P(gQ;L) of primitives. From 
equation (10), elements of the form B+(t), where t is a rooted tree, are primitive. 
We call such elements "primitive trees": they are those rooted trees whose root has 
exactly one child. If we let Tg" be the set of primitive trees (graded, like 9", by the 
number of non-root vertices), then we have the following result (for another proof 
see [10, Theorem 4.1]). 

Proposition 4.2. The vector space k{Tg"} 
(vL). 

Pro@ Since 

generated by the primitive trees is 

B+(tl) o B+(t2) = B+(tlt2) + B+(B+(h) o t2), 

k{Tg"} C T(gQ;L) is a sub-Lie-algebra. Also, since B+ is an isomorphism of 
k{9",_} onto k{Tg",}, we have dimk{Tg",} = T,-I. Then the Poincaré-Birkhoff- 
Witt theorem implies that the universal enveloping algebra of k{Tg"} has the saine 
dimension in grade n as does the symmetric algebra on k{Tg"}: but in view of 
equation (5), this dimension is T, = dim(g£aL),. Hence k{Tg"} = T(g£CL). [] 
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Suppose t, t2, t3 are rooted trees so that It I+lt21 = It31. If there is an elementary 
cut C of t3 so that 
(11) pC(t3) = tl and Rc(t3) = t2, 
let re(t1, t2; t3) be the number of distinct ele,nentary cuts C of t3 for which equations 
(11) hold: otherwise, set m(tl,t2;t3) = O. If (and only if) m(tl,t2;t3) # O, it is 
also true that t3 can be obtained by attaching (via a new edge) the root vertex of 
ri to some vertex of t2: let (tl,t2; t3) be the number of vertices of t2 for which 
this is true. Evidently, 
n.(-,t2;t3) = n(t2;t3) and m(.,t2;t3) = m(t2;t3) 
for trees t2 <l t3; so we have generalized the multiplicities of §2. (The reader is 
warned that n(h,t2;t3) as used in [3] and [5] is our m(tl,t2;t3).) We now show 
how symmetry groups can be used to relate the two multiplicities. 
Proposition 4.3. For rooted trees t,t2,t3 with ]t] + It=l -- It31, 
ISG( t, )[[SG( t2 )D,.(t, t2; t3) = 
Proof. For any rooted tree t and v  I "(t), let Fix(t,, t) <_ SG(t) be the subgroup of 
SG(t) that holds t, (the subtree of t with v as root) pointwise fixed. We can assume 
there is an elementary cut C = {e} of t3 so that equations (11) hold (otherwise 
both sides of the conclusion are zero). If e has source v and target w, then t, is 
isomorphic to t. Also, if Orb(e, t3) is the orbit of e under SG(t3), then 

m(t,t2;t3) = I Orb(e, t3)l = ISG(t3)/Fix(t.,t3) x SG(t)l = 

On the other hand, since /C(t3) is isomorphic to 

n(tl,t2;t3) = I Orb(v, Rc(t3))l = ISG(Rc(t3))/Fix(v, RC(t3))I = 

IC(t3)l 

I Fix(t,t3)llSG(t)l 

I Fix(v, Rc(t3))l" 

Since there is an evident identification of Fix(t,,, t3) with Fix(v, Rc(t3)), we have 
ISG(t3)I m(t''t2;t3)= ISC(t=)I 
and the conclusion follows. [] 

We can now use the inner product on Kreimer's Hopf algebra K to define an 
isomorphism of L onto the graded dual of K- 
Proposition 4.4. There is an isomorphism y  9{CL --' 9{K  defined by 
(X(t),u> = (B_(t),u) = (t,B+(u)) 
for any rooted tree t and monomial u of K. 
Proof. Since K is locally finite, it suflïces to prove that  is an injective homo- 
morphism. We first show ( is a homomorphism, i.e., that 

<X(tl o t2),u> = <x(tl)® 
= E(X(tl),U')(x(t2),u" } 

= E(t,B+(u'))(t2,B+(u")) 
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for any monomial u of 9çK with coproduct 
(12) () = ' ®". 
t 
In view of Proposition 4.2, 9£CL is generated as an algebra by the primitive trees. 
So it suffices to show that 
(13) (x(B+(t)ot2),u)= E(B+(t),B+(u'))(t2, B+(u"))= -(t,u')(t2, B+(u")). 
Now from the definition of n(tl,t2; t3), 
<x(B+(t) o t2), ) = (B+(t) o t2, +()) 
= E n(t, t2;t3)(t3, B+(u)) =n(t, t2;B+(u))]SG(B+(u))]. 
On the other hand, if A(u) is given by equation (12), then 
(14) A(B+(u)) = B+(u) ® 1 + -u' ® B+(u") 
t 
by equation (6). Now the only nonzero terres of 
E(t,u')(t2, B+(u")) 
are those with u' = t and t2 = B+(u"): and (comparing Proposition 3.1 with 
equation (14)) there are m(t, t2; B+(u)) such terres. Hence 
E(t, u')(t2, B+(u") ) = re(t, t2; B+(u) )ISG(t)IISG(t2)I, 
and equation (13) follows from Proposition 4.3: thus, X is a homomorphism. 
Now suppose v = i aiti E ker X- Then 
<x(v), ) =  a,(ti, +()) = 0 
i 
for all monomials u of 9£K. But setting u = B-(ti) implies that ai = 0 for each i; 
so v=0. [] 
Remark. In [18, Prop. 2.1] (and also in [9, Theorem 14.16]) it is wrongly asserted 
gr 
that the map sending B+(t) to Zt induces an isomorphism of 9fOL onto 9çg: the 
error is due to a failure to distinguish the multiplicities n(tl, t2; t3) and m(tl, t2; t3), 
since Panaite confuses the coeftïicients in 
[Zt,,Zt2] = E (m(tl,t2;t3) - m(t2,t;ta))Zt3 

It31=ltl+lt=l 
with those in 
B+(tl) o B+(t2)- B+(t2)o B+(t2) = 

In fact, since 

E (n(t,t2;t3)- n(t2,t;t3))B+(t3). 
It31=ltl+ltl 

<X(B+(t)), u) = (t, u) = ISG(t)l& = ISG(t)l<Z. u), 
we bave X(B+(t)) = ISG(t)IZ. 
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We C&ll lise the isonlorphislll ( to express the duals of P and N as maps of -GL 
(cf. Proposition 3.6 above). 
Proposition 4.5. 1. The map \-IP* X " 9£GL --, 9£GL is lefl multiplication by 
B+(.), i.e., -P*(t) = N(t)= B+(.)ot. 
2. For rooted trees t, .-N*k'(t) = (t)- B+ B_(t). 
Pro@ Using Propositions 2.3, 3.2, and 3.3, we have for any rooted tree t and 
monolnial u of «, 
(k(t),P(u)) = (t,B+P(u)) = (t,B+(u)) = ((t),B+(u)) = (X((t)),u) 

and 

(x(t),N(u)) = (t, B+N(u)) = (t, mB+(u)) - (t,B_ 
= (q3(t),B(u))- (\ç,-ooOB(t),u)=( X (q3(t)-B-,Oo;B(t)),u)  
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CONNECTIONS WITH PRESCRIBED 
FIRST PONTRJAGIN FORM 

MAHUYA DATTA 

ABSTRACT Let P be a principal O(n) bnndle over a C ° manifold I of di- 
mension rn. If n >_ 5rn + 4 + 4(m4+1), then we prove that every differential 
4-form representing the first Pontrjagin class of P is the Pontrjagin form of 
some connection on P. 

1. INTRODUCTION 
Let P be a principal O(n) bundle over a C m nlanifold /I of dimension m, and 
let Pi E t-I4i(1I) denote the i-dimensional Pontrjagin class of P. We address the 
question whether a 4i-form representing the class Pi is a Pontrjagin form of some 
connection Oll P. In [1] we considered the top-dimensional Pontrjagin class pa of a 
principal O(n) bundle P over a 4d-dinlensional open manifold M for n >_ 2d, and 
we gave a holnotopy classification of connections a on P that satisfy pa(a) = 
where a is a volume form on iii. In this paper, we take llp the case of the first 
Pontrjagin form and prove the following result. 
Theorem 1.1. If n >_ 5fa + 4 + 4(m4+1), then every dijïferential 4-form represent- 
ing the first Pontrjagin class p is the Pontrjagin fon of some connection on P. 
Moreover, when M is a closed manifold, the same is truc for n >_ 5m + 4(). 
m! 
Here () denotes the integer 
We observe that when n > m, then P reduces to the direct sum P1 @ P2 of 
two principal bundles, where P1 is an O(m) bundle and P2 is the trivial O(n - m) 
bundle on M. Since the Pontrjagin form is additive, the above observation reduces 
the problem to finding a connection on a trivial principal bundle with a given exact 
form as its Pontrjagin form. 
Now, if an exact 4-form on iii can be expressed as the sure of q prinlary mono- 
mials of the form dfl/ dr2/ dr3/ dr4, where the fi's are smooth functions on 
then we can explicitly construct a connection on the trivial principal O(2q)-bundle 
over M by taking a 2 x 2 block 
( 0 fldf2-f3df4 ) 
a= -fl df2 + f3df 0 
along the principal diagonal for each such monomial. It can be seen easily that the 
Pontrjagin form of such a connection is the given exact form on M. Indeed, we can 
prove the following result (compare ([2], 3.4.1 (B'))). 
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Theorem 1.2. Every exact 4-form dw on III can be expressed as the sure of q 
primary monomials for q >_ 2(m + 1) + 2('4+1). Furthermore, if III is a closed 
manifold, then the saine is truc for q >_ 27n n u 2('). 
In view of the above discussion it is easy to sec that Theorem 1.1 follows from 
Theorem 1.2. 
We employ the sheaf-theoretic and analytic techniques of the theory of the h- 
principle [2] to prove the al)ove result. We observe that an exact 4-form dw can 
bc expressed as the sum of q primary monomials if and only if there is a map 
f : III ---* ]l 4q such that 
q 

where r is the canonical volume form on IR 4 and ri : III ---* IR 4, i = 1, 2 ..... q, are 
components of f. The nlaps characterized by the above equation are solutions to 
a certain first-order partial differential equation. The associated partial differential 
operator being infinitesimally invertible on an open subset, we apply Gromov's 
formulation of the hnplicit Function Theorem in the infinite-dimensional setup to 
nake way for the sheaf techniques. 
In Section 2, following Gromov [2], we briefly describe the notion of infinitesimal 
inversion of partial differential operators and state some results relating to the solu- 
tion sheaf of infinitesimally invertible operators. We shall assume that the reader is 
familiar with the language of the h-principle, in particular with the terms: partial 
differential relations, holononlic section, the h-principle, (micro)flexible sheaf and 
sharply moving diffeotopy. For a brief review of terminology and sheaf techniques in 
the h-principle we refer to the AppendLx of [1]. In section 3 we consider immersions 
in a manifold N with a fixed k-form r and prove the h-principle for "r-regular" im- 
mersions that induce a given k-form on the source manifold. This has been shown 
by observing that the relevant differential operator is infinitesinmlly invertible on 
the space of "r-regular" immersions. In section 4 we prove that if (N, r) is the 
q-fold product of the k-dimensional Euclidean space with canonical volume form, 
then r-regular immersions are generic for q sufficiently large. Using genericity of 
¢r-regular nmps, we then prove the second part of Theorem 1.2. Finally, by applying 
the h-principle of a-regular maps (Section 3), we prove the full form of Theorem 1.2 
and the main result of this paper. 

2. INFINITESIMAL INVERSION OF DIFFERENTIAL OPERATORS 

Let ¥  III be a C  fibration and G  III be a C  vector bundle over a 
manifold M. We denote by 2( « and Ç« respectively the spaces of C « sections of X 
and G with the fine C « topology, for a = 1, 2,..., oe. Let/3 : ,¥ ----, ç be a C  
differential operator of order r, so that if x is a C «+ section of X, then/?(x) is a 
C « section of G for a = 1, 2 .... , oe. 
Let T«t(X) C TX denote the subspace of vertical vectors (i.e., tangent to the 
fibres of the fibration X  III) in the tangent bundle TX of X. For a section 
x : M  X, we denote the induced vector bundle x*T«,t(X) by Y. When x 
is C , this bundle is C-smooth for ]3 _< a and we denote OEv :Yç the space of 
C  sections of this induced bundle. The space :Y can be realized as the infinite- 
dimensional tangent space of 2(  at x. We define the linearization L of the operator 
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/9 at x as follows: 

d 
where {xt " t >_ 0} is a 1-parameter family of sections of X with x0 --= x and 
dxdt It=o = Y. Clearlv,. Lx is a linear diffcrential operator of order r lu y and L(x, y) 
Lx(y) is a differential operator of order r in both x and y. 
Let A C X (d) be an open subset of the d-jet space of sections of X for some 
d  r. Following Gromov, we shall call such n subset an open differential relation 
of order d. A solution of A will also be referred to as an A-regular section of X. 
Let A denote the space of solutions of the relation A. Clearly, A is contained in 
X d, and A +d = A  Xa+d is an open subset of X «+d in the fine C «+d topolo. 
D is said to be infinitesimally invertible over the subset A C X if for every 
x  A there is a linear differential operator BI« - Çs + y of a certain order s 
(independent of x) such that the following properties are satisfied: 
(1) The global operator 
al " A d x Ç  T(,t "°) 
is a differential operator that is given bv a C  map A  G ()  Tt(X), 
where T(X °) denotes the tangent bundle of 
(2) L(x, M(x,g)) = g for all x  A d+ and g  Ç+. In other words, I« is a 
right inverse of L. 
The integer d is called the defect of the infinitesimal inversion M ([2], 2.3.1). 
We now state an infinite-dinmnsional Implicit Function Theorem due to Gromov 
which generalizes Nash's theory in the context of differential operators. 
Let D be a C oe differential operator of order r. Suppose D admits an infinitesimal 
inversion of order s and of defect d. Let us fix a Riemannian metric on M. Let 
a > max(d, 2r + s). 
Theorem 2.1 ([2], 2.3.2). For every x  A  there exists a fine C + neighbourhood 
 of the zero section in the space Ça+s and an operator DZ 1 " «  A a such 
that 
(1) >ç(0) = x. 
(2) (Inversion propey) D(D(g)) = D(x) + g. 
(3) gg e  is Ce+*-smooth. dot   a, ten Vç(g) is Ce-smooth. 
(4) (Locality) The value of D2l(g) af any point v  M does not depend on the 
behaviour of x and g outside the unit ball B.,.(1) in I with centre v relative 
fo the fixed metric on I. 
In paicular, the operator D  A oe  Çoe is an open map in the respective fine 
C oe topologies. 
It is to be noted that the local inverse D21 depends on the Riemannian metric 
on I. If we choose an appropriate Riemannian metric on I, then applying the 
locality property of the inverse in Theorem 2.1 we can prove 
Proposition 2.2 ([2], 2.3.2). g D is infinitesimally invertible, then the shea] of 
A-regular solutions of the differential equation D(x) = g is microflexible. 
We now consider some partial differential relations which have the saine C  
solutions, namely the solutions to the equation D(x) = g. Let 
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consist of (et + r)-jets of infinitesimal sohltions of T) = g of order et and let T/° 
be denoted as T/. Recall that x is an infinitesimal solution of D = g of order a if 
7P(x) - g has zero et-jet. Define 
R=R  p -'(A), 
where Fd+r . .Y(+) * X (d) is the canonical projection map for  _> d - r. The 
rdations  have the saine C oe solutions for all   d-r, namely the C oe solutions 
of the equation (x) = g in M (such a solution, from now on, will be referred to as 
an A-regular solution of the equation). 
Let ¢ denote the sheaf of A-regular solutions of the equation (x) = g with 
the C oe compact open toplogy and let q be the sheaf of sections of  with C ° 
Colnpact open topology. If is a consequPnce of Theorem 2.1 that 
Proposition 2.3 ([2], 2.3.2). If  2 lll(d + 8,r+ 28), th a infinitesimal 
solution of  can be d@rmed to a local solution. Furthermore, the map J  
  , defined by .1(¢) = ç+, is a local weak homotopy equivalence. In other 
words,  saiisfies the local -principle. 

3. THE h-PRINCIPLE OF ISOMETRIC (Y-REGULAR MAPS 

We start with the following definition. 
Definition 3.1 ([2], 3.4.1). Let (N,a) be a smooth manifold with a closed k-form 
. A smooth map f " BI --- N is said to be a-regular if for each x E BI, the map 

I,,. TI(x)N - Ak-I(TM), 
ô -+ f*(ô.a) 

is surjective for all x  BI. 

A a-regular map is necessarily an illmlersiolL 
Let w be a given k-form on M for k > 2. We call a nlap f  (/I, w) 
isometric if f*a = w. In this section we shall prove the h-principle for a-regular 
isonletric nmps (M, w) --- (N, a) in the following situation: 
(1) both a and w are exact; 
(2) BI = M0 x 
(3) w is induced fronl a k-fornl on M0 by tlle projection lnap p " BI0 ×  + I0. 
Let D  C(M,N) ---, fk(Bl) dellote the first-order differential operator on 
the space of C nmps f " BI + N with values iii the space of k-forms 
defined by D(f) = f*a. Since a is a closed form, the sheaf of solutions of 
is not microflexible ([2], 3.4.1). Now, suppose that a = da and w = OE' for some 
(k - 1)-forms a and w on N and BI respectively. If f is a smooth immersion 
such that f*a = w, then locally on any contractible set the above equation reduces 
to f'al + de = 031 for some (k - 2)-form ¢ on BI. Conversely, if (f, ¢) is a pair 
satis ,fying f*a + de = wl, then f*a = w. Let 
f?'C(M, N) x f-2(B/)  Qk-l(/l) 
denote the differential operator defined by/)(f, ) = f'al +de, where f  BI -, N 
is a smooth map and  is a differential (k - 2)-form on BI. Note that the pairs 
(f, ) can be realized as sections of the fibre bundle 
which will be denoted by E for future reference. 
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The linearization L(I, ) of the operator 79 at (f, ¢) cm be obtaned as fol- 
lows: Consider a smooth 1-parmneter family of sections {(ft, et)} in E such that 
(f0,¢0) = (f,¢)- Then 
dg? 
L(L¢)(0, ) = f*d(O.al) + f* (O.d«l) + d@, 
where 0 is a vector field on N along f and  is a (k - 2)-form on M. The equation 
L(I,) = Wl can be solved for (0, ) if the following system h a solution: 
f*(O.dal) = Wl, 
/*(0.«1) +  = 0. 
Now the above systeln of equations is solvable for (0, ) if f is a a-regular map. 
Thus the operator  is infinitesimally invertible on all those (f, ¢) for which f is 
a-regultr ([2]). Since a-regularity is an t, pen condition and depends only on the 
first jet of a mai», the space of pairs (f, ¢) for which f is a-regular corresponds to 
the solution space of an open differentiM relation A C E (1), where E (1) is the 1-jet 
bundle of sections of the libre bundle E mentioned above. Hence the operator  
bas the zeroth-order inversion (i.e., s = 0, where s is defined ms in Section 2) with 
defect d = 1. 
Let  be the sheaf of a-regular solutions of the equation D(f) = w and let  be 
the sheaf of pairs (f, ¢) satisfying the equation  = w where f is a-regul. There 
is a canonicM map    that tes a pair (f, ¢) onto f. Furthermore, (x) h 
the saine holnotopy type as the space (x). 
Let  C E (+1) consist of ( + 1)-jets of infinitesim solutions of  = Wl of 
order  and let  = (pî+l)-(A), where pî+l . E(+) « E() is the cmmn- 
icM projection. The following proposition is a direct consequence of Proposition 2.2 
and Proposition 2.3. 
Proposition 3.2. (i) Th« solution sh«af  of  = w is microfl«xibl«. 
(il) The 3-jet map j3: (x)  ç(x) is a weak homotopy equivalence for evew 
infinitesimal solution of order 2 of  =  where f is also a-regular, then (f, ¢) 
can be homotoped to a local solution of the equation. 
Theorem 3.3. Let a be an exact k-form on N as above and let M = Mo x . If 
the form w = &t on M is induced from an exact k-form on Mo by the projection 
map p : kIo x   kIo, then every section of 2 i8 homotopic to a holonomic 
section (in the space of continuous sections of 2 with C ° compact open topology). 

Proof. Let  denote the sheaf of sections of the jet bundle /(3) with images in 2- 
We shall prove that 
is a weak homotopy equivalence. 
First observe that the fibre-preserving diffeomorphisms of M0 x ]R act on the 
sheaf . To see this take a smooth ilnmersion f : M0 x ]R  N and a (k - 2)-form 
¢ such that f*«l + de = Wl, where Wl = p*wo for some (k - 1)-form on M0. Let 
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6  Mo x 1R -, Mo x 1R bc a fibre-preserving diffeomorphism so that p o 6 = p. 
Define the action of 6 by 
&(f,ç)(fo6,6*ç). 
Then, 
(f  )*« + d(*¢) = *(f*« + de) = 6" = *p*0 = p*0 = . 
Also, if f is «-regular, then so is f  «. 
On the other hand, the fibre-preserving diffeotopies sharply move 110 in I0   
([2], [1]). Since the sheaf ç is mi«roflexible (Propositi,m 3.2), we conclude that the 
restriction of  to M0 is flexible ([2], 2.3.2,[1]). 
A standard argument provcs that lhe sheaf  is flexible ([2], 1.4.2 (A')) and 
Proposition 3.2 (il) says that 
j  ((.r)  (z) 
is a weak holnotopy cquivMence for every x G M. Theu by the Sheaf Homomor- 
phil,, çl,ern, ([2], 2.2. )) 
is a weak homotopy equivalcwC. 
ina.lly, the theorcm follows froll thc observation that M0   can be deformed 
into an arbitrary small neighbourhood of M0 by means of fibre-preserving diffe 
morphisms of M. 
Let  consist of 2-jets of «-regular infinitesimal solutions of order 1 of the 
equation D =  and let F() dcnote tire spe of continuous sections of  with 
C ° compact open topolo. Then we have the following. 
Corollary 3.4. An arbitrary section of  C J2(I. N) is homotopic fo a holo- 
nomic section in F(). Hence, the «-regular isometric C oe imersions f " (l&  
, d = do  O)  (N, d) satisfy the h-pri'nciple. Furthermore. if I is an 
open manifold, lhen «-regular isolropic immersions satisfy the h-principle. 
Proof. Let (f, ¢) be a second-order infinitesilnal solntion at x of the equation  = 
. Then " 
Ak(M) () associated to the exterior differential operator d such that A_(j) = 
Jd" Then applying A_ on the precediug equation we get j}.«(x) = j(x). Thns 
f is an infinitesimal solution of order 1 of the equation D = . Hence we bave the 
canoni«al nmp p" 2  ni that maps (j(x),fi(x)) onto j(x). X shall prove 
that this map is surjective and that fibres of p are affine snbspaces. This would 
imply that p bas a section, and then the first part of the corollarv wonld follow 
from the above theorem. 
To prove that p is surjective, consider the followiug sequence of vector bundles: 
...--+ (A"-2(I)) (3) A- (Ak-(,l)) (2) ' (A(5I))(1) ... 
where the bundle maps k are induced by the exterior differential operator d  
-i--1 
 o j  3d  By the formal Poincaré Lemma this sequence is exact. 
Let f be a first-order infinitesimal solntion of  =  at x E 51, which is also 
«-regnlar, so that jî(x)  . Then j).«(x) = j(x) and consequently Jî*« (x) - 
j (x) is in ker _. Hcnce there exists a 3-jet j(x) snch that 
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Therefore, (jÏ(.r),j(x))  7 and p is surjective. 
Now let Jî(x)  7. Then p-l(jy(x)) consists of all pairs (fi3('),j(x)) e E (3) 
su«h that j(x) = Jî(x) and j¢(x) 
- 
-] Y .2. -2 
 1 2 ( g   (  ) I  (x)). This shows ht the fibres of p are ane subspaces and 
that p : 2   is an anc bundlc. This proves thc first part of the corollary. 
To prove the second prt, ont has to note, in ddition, that tlhe zcro fonn 
iwdant undcr anv diffeomorphism of M, and M cm be dcfonned into an arbitry 
small neighbourimod o its (m 1 1)-skclcton by 

4. EXISTENCE OF r-REGULAR IMMERSIONS INDUCING w 
Let r0 be a clused k-fonn on a lnanifold NO, and let N be the q-fold C&rtesi&n 
q 
product of N0 with the '-fornl ff = i=1 gff0, where wi " N  N0 is the 
projection onto the i-th factor.  first deterlnine when the g-regular nl&ps exist 
generically and then prove the existellCe of isometric nmps, applying the results 
obtained in the previous sections. 
Definition 4.1. An imlncrsiou f = (ft,f2,...,fq) : AI  N is said to bc g0- 
large if f«o,..., Jq«o span t.he '-th extcrior bmldlc Ak(M); this lnoans, for every 
-forln w OIl I. there exist continuous functions Pi : AI  N, z = 1 ..... q, such 
that 
q 
Let 
A= {(e,, .... eq) e 4(M,N) ' e[«o,...,eq«O span Ak(TxM),x G M}. 
If f = (f,---,L) is a solution of A, then H«o(X) .... ,f«o(x) span Ak(TM) for 
eh x  M. Moreover, it follows froln the lemma below that the «0-large maps 
are precisely the solutions of the relation . 
Lemma 4.2. Let t,...,Wq be k-forms on M such that for eachx G M, (x),..., 
q(X) span Ak(TxM). Then Wl ..... Wq span the space of k-forms '(I) over the 
riw of continuous functions on M. 
Since  is an open relation, the «0-large immersions form &l open set in the fine 
C  topology. Next we observe that 
Proposition 4.3. If f = (fl,..., fq) : M  N is a «o-large immersion, then f 
is «-regular. 
Pro@ Let f = (f, .... fq) be a «0-large ilmnersion of Al into the q-fold product 
of N0. If 01,02,... Oq are vector fields on M, then we have the relation 
q q 
i=1 
where i = (fi).Oi is a vector field on N along fi- The proposition now follows 
from the following simple observation. 
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Lemma 4.4. 
linear map 

is surjective. 

If COl, .... COq are hnear k-forms on IR ' spanning Ak(IR'), then the 
Nrn X-'" X lI m ---+ flk(Nm-1), 
q 
01,... , Oq  Ei=I Oi.COi 

Iu the rest of this article, /il and (N, er) will be as follows: 
(1) 
(2) N will denote the q-fold Crtesian product of the Eucli(lean space 
(3) er will denote the/,'-form obtained by summing the q canonical volume forms 
erk := dyl A.  .Adyk on each IR  factor, where Yl, Y2 .... , y are the cauonical 
coordinates on IR k. 

Proposition 4.5. If q > m + ('.), then fer. .... ,ferk span the k-th ex.terior 
buMle of 51 for generic (f, .... fq) " 51 ---+ N. Corsequently, "zf q > 2m + 2(), 
there exists a erk-large irwmersion f :ii[  (N, er) such that ff(er) = O. 

q 
Pro@ Hcre N = IR qk and er = $i=1 °'k" Fix a basis el, e2 .... , e, for IR TM. Let L 
be a lilmar map froin IR ' to IR @. Theu L can be expressed as L = (L, L2,..., Lq), 
where Li is the projection of L onto the i-th copy of 
If L is erg-large, then the forlllS Lierk, LE k .... , Ler span the bundle A(IRm). 
Note that the k x k cofactors of Li correspond to the values of Ler on the k- 
tuples of basis vectors (%,... ,e), where {i1,i2,---,i} is an ordered subset of 
{1, 2 .... , rn}. If i deuotes the column vector foriued by the k x k cofactors of the 
natrix Li, then by a er-large coudition on L is nmaut that Ê = (1,--., ]q) has the 
maxinmm rank. Let ' consist of all linear luaps L 
such that rank Ê is strictly less thau l = (k), iii other words, any l x l cofactor of 
/_, is zero. Therefore, ' is senfialgebraic aud hence stratified ([2], 1.3.1). Moreover, 
the oimenion of Z' iu (Nr,N)is q- () + 1. 
Let  be the subset of the 1-jet space J (/il, N) consisting of all 1-jets j}(x) such 
that {fer  i = 1,2 .... ,q} do llot span Ax(5l). Hence a inap f " 5I ---+ N is 
era-large if its I-jet map misses the set E. Siuce er has global symmetry, the singular 
set E in the I-jet space fibres over I and therefore it is stratified with codimension 
q - () + 1. Hence by the Tholn Transversality Theoreln, a generic lnap is erg-large 
if q- (.) _> m. 
Now, let f = (f,..., fq) : Iii  IR @ be a erg-large ilnmersion: then define 
/---- (/1,''', fq) as follows: 

/i m (fi2,fil,fi3,''',fik), 

where fi = (fi, ri2, ri3,--., A) : Iii --- IR . Note that f'erk = --f'erk for every i. 
Hence (f, f) : 211  N q X N qk is a erg-large immersion of Iii into lI 2qk that pulls 
back er @ er onto the zero form on 211. [] 

Theorem 4.6. Let M be a closed manifold. If q _ 2.m+2(), th.en every exact form 
on M can be induced flore er by a er-regular immersion f : 211 ---+ N. Consequently, 
every exact k-form on a closed m-dimensional manifold is expressible as the sure 
of q primary monomials for q >_ 2m + 2(). 
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Proof. Let T = Yl dy2 A ... A dyk so that «k = dT. It follows from Section 3 and 
Proposition 4.5 that the operator 
q 
) " (fl, f2, . . . , f, ¢)  -,:T + de 
is infinitesimally invertible on a-large immersions which exist generically for q _> 
m + (). Hence by Theorem 2.1, the image of ak-large immersions under  is a 
nonempty open set in the fine C  topology for q >_ m -t- (). Moreover, when 
q >_ 2m + 2(), there exists a ak-large imlnersion f = (fl,...,fq)" M  I kq 
such that q * 
-=1 f ak = 0, which implies that q * 
-i=1 f T is a closed form. As a 
consequence, hnage/) contains a closed (k- 1)-form c. 
Let M now be closed and let w = da be exact. Then for suflïciently small A > 0, 
c+ At E hnage/). In other words, there exists a ak-large immersion (gl, g2 .... , gq) 
and a (k - 2)-form ¢ such that 
q 
c + As =  g; + de, 
i=l 

and therefore 

Clearly, ( 1 1 
---gl, --g2, -   
the theorem. 

x-- q 1 
w = 2_,(----- g) 
i=1 
"'VA 
 g 
, « / ) is a-regular and this completes the proof of 

The next result is an immediate consequence of the above theorem. 
Corollary 4.7. If ]lI is arbitrary, then every compactly supported exact k-form on 
M can be induced by a a-regular immersion f" M  (N,a) for q _> 2m + 2(). 
Corollary 4.8. If P is a principal O(n) bundle over a closed manifold M, then 
every compactly supported 4-fo on I representing the first Pontrjagin class of 
P is the Pontrjagin fo of some connection on P, for n  5m + 4(). 
The proof of the above corollary will be similar to that of Corollary 4.10 and we 
omit it here. 
Theorem 4.9 ([2], 3.4.1 (B')). Let (N,a) be the q-fold product of (k, ak) for 
k  2. Ifq  2(m+1)+2(:), then an arbztrary exact k-fo onM can be 
induced by a a-regular immersion f  M  N. Therefore, every exact k-fo on 
an m-dimensional manifold is expressible as the sure of q primary monomials for 
qoE2(m+l)+2( k ). 
Proof. Let x, x2,..., x denote a local coordinate system on M. Then a k-form 
 on M can be represented IocaIIy as: 
  idoel 
I 
where I runs over all multi-indices (i,i,...,i) for 1 N i < i < --. < i N m, 
l = il,i ..... i are smooth functions defined locally on M, and dz = dxi  dzi  
""  dxi«. 
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q 
Recall that « = @i=1 k, where er k is the canonical vohline fornl on k. If 
f = (f .... ,fq  kl + q is a sinooth map, then f*« = w defiims for each 
nmlti-index I = (il,i2,..., ik) with 1 <_ il < i2 < -'" < ik <_ m ail equation EI: 

where c represents all eleinent of the symmetry group Sk on k letters {1.2 .... , k}, 
and {fja} «enote the COnlpoi,ents of fj. 
Differentiating Et with respect to xp, p (E {1,2 ..... m}, we get an equation E: 

q 
E E (-1)sgn 02fJ'l Ofj,2 Ofj,k Ow, 
j= r,c O.rpO.ri( O.ri( "" O.ri OXp' 

we obtain a systein of equations 
(il,i2,--.,ik) with I < il < i2 < -- 
system of equations is independent of 
relation l in the 2-jet space. 

 Êï}, where I runs over ail nmlti-indices 
 < ik < m. It can be verified that this 
coordinate transformation and defines the 

Note that {Ê} is a system of m() equations that are linear in the variables 
Vipa, the total number of which is kqm(m + 1)/2. Let A denote the coefficient 
matrix of the vector {viÇ  } in the system {Ê}. The systmn of equations {Êï} bas 
a solution if the matrix A bas the maximum rank everywhere. Since k > 2, the 
condition "rank A < maximmn" defines a stratified subset E in the 1-jet space 
.l(M,lq). If q is such that kq(m + 1)/2 >_ () + 1, then codim E > m and hence 
by the Thom Transversality Theorem, j} misses E for generic f. In other words, 
we get a lnap f for which the following system of equations bas a solution for each 
x  ]Il: 

(1) 

q 
E E (-1)sgn Of'm2 
j=l r,c, OXi'r(2) 

Moreover, the space of solutions is an affine subspace in IR d of codimension m(), 
whee  = q,(m + 1)/2. Theefore, if  > (1) +  + 1, then there exists 
a map f  ]Il x IR - (N, «) that is a «k-large immersion and for which the 
J -J Since f is «k-large, there 
system of equations (1) has a solution, say vip = v w . 
q * 
exist continuous real-valued flmctions/i on/11 such that w = i=; iLi °-k, where 
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Li denotes the derivative map dfi. Define for each i = 1,2,...,q, a bundle map 
Li  T11I  T]R k by 
{dfi(x) if/3i(x) > 1, 
L(x) = dfi(x) if/3i(x) < 1, 
where ri is obtailmd froin fi by interchaiging the first two component functions. 
Take T = (L1 .... ,Lq,lL1,...,qLq), where  = I - ii t. Note that T extends 
continuously over ail of M if we define it t.o be identically zero on the set 1(1). 
Thus we get a a-regular bmadle nmp T  TM  T q such that T*a = w. We 
Ja -a forj < and ja =0 
extend this (locally) to a section of  by taking Yip = Yip -- q 
for j > q. These local solutions finally define a global section ofgl ifwe patch them 
together by a partition of unity. (Note that the system of equations (1) is line in 
Vip .) We now conclude the existence of an isometric immersion by Theorem 3.3. 

Theorems 4.9 and 4.6 prove Theorem 1.2. 
Corollary 4.10. Let P be a principal O(u) bundlc over a manzfold 11I of di'mension 
m, and let n >_ 5m + 4 + 4(m4+1). Then every 4-form on M representing the first 
Pontrjagin class of P is the Pontrjagin form of some connection on P. 
Pro@ If n > diln/I, then P can be reduced to P @ P2, where P1 is a principal 
O(m) bundle and P2 is the trivial O(n - m) bundle over M. This may be seen 
easily if we view a principal O(n) bundle as a fralne bundle associated to SOlne vector 
bundle of rank n. Moreover, we have a canonical inclusion Q = P @ P2 - P 
that takes the fibres of P @ P2 canonically into the fibres of P. Now we prove that 
the Pontrjagin forms of the bundles P raid Q are the sanie. It is a standard fa.et 
that a connection OQ on Q can be extended uniquely to a connection O:p Oll P such 
that i*ap = aQ. We shall show that p(aO. ) = pl(p). Vre recall that the first 
Pontrjagin form p (aQ) is uniquely determined by the equation 
(2) 7r pl(Q) = trace (Dc 
where D stands for the covariant differeltiation and rrQ denotes the projection map 
Q  /lI. Similarly, rr},p(ap) = trace(Dap A Dap) ([3]). Taking pull-back bv 
i we get i*rc*ppl(ap) = trace(Daq A Daq). Since rrp o i = rrQ, the left-hand 
side is equal to rrQpl(ap). Hence by equation (2) and the uniqueness property, 
pt(ap) = p(aq). Moreover, the Pontrjagin rotin is additive, so that if 
are connections on PI and P2, respectively, then pi(al @ a2) = pl(oq) -t-pi(a2). 
In view of the a.bove observation it is enough to show that every exact, form on AI 
is the Pontrjagin form of some colmection on the trivial principal O(n) bundle for 
m+l 
n_>4(rn+l)+4( 4 )" 
Let. daa be an exact. 4-form on AI. We have proved in Theorenl 4.9 that an exact. 
4-form on a manifold of dimension m tan be expressed as the sure of q prinmry 
 Yi= dfil A dfi2 A dfi3 A dfi4, 
nonomials for q >_ 2(. + ) + 2(% +') Let, d = 2 
where fij are smooth functions on/1I and where q satisfies the above relation. Now 
consider an 0(2q)-valued 1-form a on ]il such that corresponding to each lnonomial 
dffl A dfi2 A dfi3 A dfi4 there exists a 2 x 2 block 
( 0 fildfi2-fi3dfi4 ) 
a, = -fiat dfi2 + fi3 dfi4 0 
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along the principal diagonal, all other elements being zero. Çlearly a is a connection 
on the trivial principal O(2q)-bundle over _bi and its first Pontrjagin form is 
q 
p,() = p,() 
q 

=Z 

q 

=Z 

trace (Dal A Da) 

trace (dai A dai) = 

i=1 
This colnpletes the proof. 
Corollaries 4.8 and 4.10 prove Theorem 1.2. 
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SELF-INTERSECTION CLASS FOR SINGULARITIES 
AND ITS APPLICATION TO FOLD MAPS 
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Dedicated to Professor Takuo Fukuda on the occasion of his 60th birhday 

ABSTRACT. Let f : M - N be a generic smooth map with corank one sin- 
gularities bctween manifolds, and let S(f) be the singular point set of f. "vVe 
define the self-intersection class l(S(f)) E H* (/1; Z) of S(f) using an incident 
class introduced by RimAnyi but with twisted coeIficients, and give a formula 
for l(S(f)) in terres of characteristic classes of the rnanifolds. We then apply 
the formula to the existence problem of fold rnaps. 

1. INTRODUCTION 
Given a smooth map g " /1I  N between smooth manifolds, does there exist 
a smooth map homotopic to g that has at most "'nice" singularities? If hot, then 
what is the obstruction? 
Embeddings and immersions, which have no singularities at ail, bave been par- 
ticularly well studied since the concept of a manifold was established around 1930. 
For example, Whitney proved that every n-dimeisional inanifold can be embedded 
in R  and iminersed in R 2'-1. Furthermore, the Smale-Hirsch theory [18] gives 
a satisfactory answer to the existence problem of immersions in terres of homotopy 
theory. Note that these results make sense only when n. : dira/1I < dira N = p. 
On the other hand, similar problems for the case n _> p seem to have been hardly 
studied except for maps of open manifolds (for example, see [22]). If/1I is closed 
and n _> p, then every smooth map f  ]il  N into an open manifold N must 
have singularities. In this paper, we consider fold rnaps -- smooth maps that have 
at most fold singularities (for details, see §4) - for such cases, and discuss their 
(non)existence problem. Note that a fold singularity is the simplest of the generic 
singularities: for exainple, a fold map for p = 1 and N = R is nothing but a Morse 
flmction. 
When p : 2, the existence problem of fold maps has been solved: a smooth 
map of a closed connected n-dimensional manifold 11I with n _> 2 into an orientable 
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surface is homotopic t.o a fold nmp if and only if the Euler characteristic of 31 
is even [31], [20], [9]. Recall that generic maps into surfaces have fold filial cusp 
singularities, and that the munber of cusp singularities has the saine parity as 
thc Euler characteristic (or equivalcntly, the top Stiefel-Whitney class wn(]il) E 
H'*(M: Z2) = Z2) of the source manifold. The above result shows that this is the 
ouly obstruction for the existence of fold lnaps into orientable surfaces. In fact. 
out pl'o|)lem is closely related to the coucept of the Thom polynomial of a given 
singularity type. as explaincd below. 
Let E be a singularity type, which means a certain talion of orbits of the group 
Diff(R ', 0) × Diff(R », 0) with respect to the right-left action on some jet space 
,l»(n,p). For a gcncric map f  bi -- N betwecn smooth manifolds, E(f) C ]il 
dcnotes the singular point set of f of type E. It is known that for E good enough. 
the closurc E(f of E(f) rcprescnts a homology class and that its Poincaré dual, 
dcnotcd by Tp(E)(f), can bc dcscribed as a polynomial in characteristic classes of 
T]II and f*TN. This is callcd the Thom polynomial for E and does hot depend on 
a particular ('hoice of f (foi" exanlple, see [17], [3], [24], [12]). Clearly, if Tp(E)(f) 
or Tp(E')(f) for some E' adjacent to Z (i.e., E' C E \ E) does hot vanish, then 
there is no generic mai) f holnotot)ic to f such that E(f) = 0. However, in general, 
even if Tp(E)(f) vanishcs, we CalmOt con«lu(le that the corresponding singularity 
Cml be eliminated bv homotopy. An examI)le in low dimensions was discovered by 
the second author, as follows. 

Theorem 1.1 (Saeki [25], [26]). Let AI be a smooth 4-ranifold with H.(]ll: Z) - 
H.(Cp2; Z). Then there exists no fold map of]il into any orientable 3-manifoid. 

Namely, for ]il as above, every generic map f : ]il -- N into an orientable 3- 
manifold N must bave cusp singularities. Note that the Thom polynomials for cusp 
singularities (or A2-type singularities) and its adjaceut swallowtail singularities (or 
A3-type singularities) both vanish in our case. 
In the proof of Theorem 1.1, the following congruence, which the third author 
[29] proved by using a Rohlin type theorem, played an essential foie: for a generic 
map f : M  N of a closed oriented 4-mauifold ]11 with HI(M: Z) = 0 iuto an 
orientable 3-manifold N, we have 

(1.1) S(f). S(f) =- 3cr(]il) (mod 4), 

where S(f) C M is the set of ail singular points of f, called the singular set of f, 
S(f). S(f) is the self-intersection nmnber of S(f) in M, and cr(]il) is the signature 
of M. In a sense, the congruence (1.1) gives us lnore information than the usual 
Thon polynomials. 
The purpose of this paper is to describe what is the essential point behind the 
congruence (1.1) and to give its integral lift for general dimensions. For a certain 
generic map f : _bi -- N between smooth manifolds whose singular point set S(f) 
is a smooth submanifold, we define the self-intersection class of S(f), denoted by 
I(S(f)), as the cohomology class in H*(]i/; Z) Poincaré dual to the homology class 
represented by the transverse iutersection of S(f) and its small perturbation in ]il. 
This class coincides with the Gysin map image of a special kind of the incident 
class introduced by Rimgmyi in his theory of Thom polynomials [24] but refined by 
using twisted coeftïcients (for more details, see §3). By using the desingularization 
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method, we obtain a formula for I(S(f)) as follows, where Ee denotes the sub- 
manifold of the 1-jet bundle j1 (AI, N) consisting of the 1-jets of kernel dimension 

Theorem 1.2. Let Iii and N be manifolds of dimensions n and p respectively, 
where Iii is closed and n-p+ 1 = 2k, n >_ p >_ 1, k >_ 1. Furthermore, let 
f  Iii -+ N be a smooth map such that jlf is transverse to E 2k and Ee(f) = 0 for 
all  > 2k + l. Then we have 
(1.2) I(S(f)) = pk(Tiil - f*TN) E H4}(iil; Z) (modulo 2-torsion), 
where p}(Tiil- f*TN) is the k-th Pontrjagin class of the difference bundle Tiil- 
f*TN. 
In the terminology of [115, Chapter VI, §1], the above condition on jf is equiva- 
lent to f being 1-generic and having corank at most one evervwhere (the corank of 
f at x is defined to be min(n,p) -tank df). In this case, we say simply that j is a 
generic smooth map with corank one singularities. Note that then the singular set 
S(f) = (jlf)-l(E2) is a smooth regnlar snbmanifold of M. Note also that this 
condition is generic provided that n > 2p - 4. 
The above theorem implies, in particular, that for a generic map f  BI -* N of 
a closed oriented 4-mauifold iii into an orientable 3-manifold N, we have 
(1.3) S(f)-S(I) = pi[M] = 3«(M) E Z, 
by the Hirzebruch signature fornmla, where p[ilI] denotes the first Pontrjagin 
number of Iii. This is nothing but an iutegral lift of the congruente (1.1). Clearly, 
Theorem 1.1 tan be proved by using (1.3). As another application of the formula 
(1.2), we have the following necessary condition for the existence of fold maps. 
Theorem 1.3. Let g  Iii -+ N be a smooth map between smooth manifolds, where 
M is closed. We assume that n = dira Iii and p -- dira N satisfy  - p + 1 ---- 2k for 
some positive odd integer k. If there exists a fold map homotopic to g, then there 
exzsts a cohomology class x  H2}(/ll; (.9TM-a.TV) such that 
x .- x = pk(Tiil - g*TN)  Hae(iil: Z) (modulo 4-torsion), 
where ,- denotes the cup product and (TM--g*TN is the orientation local system 
associated with the difference bundle Tlll - g*TN. 
The above theorem is very useful for obtaining nonexistence results for fold 
maps. For example, we will show that for certain dimeusion pairs (n,p), every 
n-dimensional oriented cobordism class contains a connected manifold that admits 
no fold ,naps into certaiu p-dimensional manifolds (for details, sec §4). 
Throughout the paper, we work in the smooth category; that is, all manifolds 
and maps are differentiable of class C  unless otherwise stated. 
The authors would like to express their sincere gratitude to Julius Korbag and 
Peter Zvengrowski for their useful suggestions. 

2. GYSIN MAP WITH TWISTED COEFFICIENTS AND SELF-INTERSECTION CLASS 

In this section, we recall the definition and some properties of the Gysin map 
with twisted coefficients induced by a smooth map between manifolds, and define 
the self-intersection class of a submanifold. 
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In the fifllowing, fol a manifold X (or a vector bundle ), Ox (resp. Oç) will 
denote the orientation local system associated with X (resp. {). 
Let f  I + N be a Slllooth map between smooth ma.lfifolds of dimensions n 
and p respectively, and £ ail arbitrary local system over N. Take an embedding 
e  M + Int D  for SOlne r and identify M with the inm.ge of the embedding 
(f,e)  M +N xD . Let U beatubular neighbourhood of M in N x D  with 
projection Pu " U  I, where U is identified with the total space of the normal 
bundle oe of BI iii N x D". Note that the orientation local system O is isolnorphic 
to that associated with t.he difference bmidle f*TN-TM, i.e., O  OI.TU_TM  
OM  f*O. 
Then, we define the (twisted) Ggsin map (or the Umkehr map) 
fi " H*(M; f*£  OI.yN_yM)  H*+(P-)(N:£) 
induced by f by the çOlnpositiol 
H*(a/; f*£  OI.yN_Tt)H*(U; (ptj)*f*£  (ptr)*Oi.yN_yM) 
H*+(P+-)(U, U x M; (Pu)* f*£) 
@l*+(P+r--n) [ A; r D" 
 ç, X D ,N X [;(PN)*£) 
1 
(iM)**H*+(P+-)(N x D r, N x ODr; (pN)*£) ((pN)*)- ,H*+(P-n)(N; 
where u  HP+'-(U, U x M; (pU)*Oi.rNlrt) is the Thom class of the norlnal 
bundle oe, ex denotes the excision isomorphism, PN " N x D r  N is the projection 
to the first factor, it " (N x D r, N x 0D")  (N x D r,  x D r x BI) is the inchlsion, 
and the final isonorphism COlnes fronl the Kiinneth theorem. It is known that the 
above definition does hot depend on the choice of a particular embedding e or r 
(for details, see [7], [8], [6], for example). 
The following properties are known. 
Lemma 2.1. (1) If f is a proper map. then the twisted Ggsin map f coincides 
with the composition 
N -1 
I. ,H_.(N:£  ON) ( [ ]) ,H*+(P-n)(N: 
whe H denotes the homolo99 of closed support, and [MI  H(M;Ot) and 
IN] e H(N; ON) are the fundmnental classes 4 M and N respectively. 
(2) We bave 
for all x  H*(N: £) and all 9  H*(M; f*£'  Oi.rN_rM), where £ and £' a 
arbitrar 9 local sgstems over N. 
(3) For smooth maps f  M  N and g  N  L with dinlM = n,dimN = p 
and dimL = g, we have (g o f) = g o f  H*(M; (g o f)*£  O(i).rr_rM )  
H *+(e-) (L; £) for an 9 local system £ over L. 
Using the twisted Gysin map, we define the self-intersection class of a submmfi- 
fold as follows. 

Definition 2.2. Let. Y be a smooth submanifold of a smooth manifold X of 
codimension .. Let U be a tubular neighbourhood of }" in X with projection 
Pu " U -- Y, where U is identified with the total space of the normal bundle vr of 
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Y in X. Consider the Thom class u E H'(U, U \ t'; (Pç)*Ovr) of uy. The twzsted 
Euler class e(oey) E H'(Y: Ov ) of oey is the image of u under the composition 
. , (r)* ,r,,,T *O 
U'(U,U \ Y;(Pu) Or)---n (t;(pu)  ) _ 
where y : U -- (U, U \ Y) is the inclusion. Then the self-intersection class I(Y)  
H2'(X; Z) of)" is defined by I(Y) = i!(e(oe)-)), where i! : H'(Y; Or) -- H2'(X: Z) 
is the Gysin map induced by the inclusion i : Y -- X. 

It is clear that when X is oriented, the honlology class Poincaré dual to the 
self-intersection class I(Y) is represented by the transverse intersection of Y and 
its small perturbation in X as a Z-cycle (even if Y is non-orientable). Note that 
this integral cycle is well-defined as a honlology class, which is denoted by Y  Y  
Hn-2,(X; Z) with n = dinlX (see [4, p. 583]). Note that Y-Y depends on the 
choice of an orientation for X, while I(Y) does hot. 

3. PROOF OF THEOREM 1.2 

hl this section, we COIlSider the self-intersection class of the singular set of a 
generic Snlooth map with corank one singularities and prove the formula (1.2). 

Proof of Theorem 1.2. Let us first assunle that N = R p. The general case will 
follow from this special case. 
Let. r  G -- /I be the Grassmamfian bundle of unoriented 2k-planes in T]iI 
and ¢ the tautological 2k-plane bundle over G. Let. us consider the vector bundle 
Hom(¢, p) over G, where P = 7c*f*TR v is the trivial p-plane bundle. There is 
a natural section s of this vector bundle associated with f, which is defined by 
s(x,H) = df,[H " H -- R v for x  M and H C T]lI. Our assmnption on 
jlf implies that s is trmlsverse to the zero section (for details, sec [23]). We set. 
(f) = s-(0) and  = r]g(l). Furthermore, we denote by j  ç(f) "-, G the 
inclusion. Note that 5 " ç(f) + S(f) is a diffeonlorphism and that the nornlal 
bundle tj of the embedding j is isonlorphic to j* Hom(¢, P)  Honl(j*¢, j*P). 
Note also that OHom(¢,«)  (O¢) ®p raid hence that Ov - j*(O¢) ®p, where for a 
local system £;, £®P denot.es t.he p-fold tensor product of £. 
Let.  be a t.ubular neighbourhood of (f) in G with projection PO" ] --' (f)" 
Furthermore, let. E denote the total space of the vector bmdle Honl(ç,P) and 
r  E + G the projection, where we consider G t.o be embedded in E as the zero 
section. Note that U tan be identified with the total space of j* Hom(¢, P). Then, 

by considering the commutative diagram 
H2kp(u, U \ ç(f); (p)*O,)  

we see that 
(3.1) 

j!(1) = e(I-lom(¢,e)) = e(¢F e H2kp(G;(O¢)®P), 

where (I) " G  (G, G \ (f)) and   E - (E, E \ G) are the inclusion maps. 
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Let K = kerdf and Q = cokerdf be the kernel bundle of rank 2k and the 
cokernel bundle of tank 1 defined over S(f), respectively. Note that Hon(K, Q) is 
isolnorphic to the normal bundle u of i: S(f)  M (for example, sec [5], [15]). 
Over S(f), we have *u  Hom(*K,*Q) and *K  j*ç. Hence, we have 
*0  *OK  j*O é, since the rank of K is even. 
Lemma 3.1. We bave 
(3.2) e(*u)  e(j*¢) e H2k((f);j*Oe) (moduto 2-torsion). 
Pro@ When 5*Q is trivial, (3.2) obviously holds even without taking modulo 2- 
torsion, since *oe  *lç  j*¢. When *Q is hot trivial, let  : (f) + (]) 
be le double covering corresponding o Wl (*Q)  Ht((f): Z2), the first Stiefel- 
Whitney class. Then, since **Q is trivial, we have e(**u) = e(*j*¢) 
H2k((f); *j*Oe). Then by Lemma 2.1 (2), we have 
2ç(*u) = (e(**u)) = (e(*j*¢)) = 2e(j*¢), 
since (1) = 2. Thus (3.2) holds. 
Lemma 3.2. We ha, ve 
(O) -2 ¢, Oa  *O» 
Pro@ Recall that the Grassmalm maniflfld Gk( 
is orientable if and only if n - 2k is evelL Furthermore, its orientation remains 
invariant under the orientation reversal of . Hence, when n - 2k is even, we have 
(O¢) -  Oo  Oa  *O» 
When n - 2k is odd, let  be the tautological 2k-plane bundle over Gk(). 
Then the orientation local system 
is isomorphic fo Hom(?, 7), where 7 is the orthogonal complement of 7. Hence, 
we have (0¢) -  Oa  
By the above lenmm, u : G + M induces the Gysin map 
 : H*(G; (0¢) -)  H*(G; Oa  *OM)  H*-("-2)(M; Z). 
Lemma 3.3. It bave 
(e(ç) "-) = 1, 
where  : H2("-)(G;(Oé)$"- )  H°(M;Z) is the G9sin map induced b 9 
:GM. 
Pro@ The proof is similar to that in Porteous' paper 
follows. 
We have only to consider the equality over a fibre G(TM,) = -(), that is, 

over the Grassmannian of T]il, since the diagram 
H 2k(n-2k) (G; (O4)®n-2k) 
(iG(Tltlx) ) " 
H 2k(-2k) (G(TM.); (O(.))*-2) 

u°(]it; z) 
H°(x; Z) 

is comnmtative, where iG(T^& : G(T]Il.) - G and i. : x --, ]il are the inchlsions, 
rr. = rl(r^& ) : G(TM.) - x, and 4)(x) is the restriction of 4) to G(TM.). 
Fix a nonzero vector v  T]ilx. Then it induces a section s, of b(x) defined bv 
s,,(H) = Iv] e TM,:/H" = H for H 



SELF-INTERSECTION CLASS FOR SINGULARITIES 3831 

where H ± is the orthogonal complement of H with respect to a fixed inner product 
on TMx. The zero set sjl(0) (= {H : v E H±}) represents the homology class in 
H2k(n-2k)-2k(G(ThIx); OG(TM:) 1 O¢(x)) Poincaré dual to the Euler class e(dp(x)). 
If we take a set of linearly independent (n - 2k)-vectors of TMx, then we get n - 2k 
generically linearly independent sections si, i = 1,2,..., n- 2k. The intersection of 
all s;-l(0) consists of a unique point, which corresponds to the 2k-plane orthogonal 
to the space spanned by the (n - 2k)-vectors. Thus we have that the homology 
class in Ho(G(TMæ); OG(TII= ) @ (O¢(x)) @n-2k) - Ho(G(TMæ); Z) Poincaré dual 
to e(çb(x)) -2k is represented by a point. Hence, the result follows by Lemma 2.1 
(1). [] 
Let us go back to the proof of Theorem 1.2. We fix a Riemannian metric on AI 
and consider the orthogonal decomposition rr*TM = çb @ çb ±. Note that e(çb) 2 = 
p(¢) (for example, see [21, §15]). By using the product formula 
p(rr*TM) = p(dp)p(dp -L ) (modulo 2-torsion) 
for Pontrjagin classes together with Lemma 2.1 and (3.1), we have 
j!(e(j*¢)) = e(çb)j!(1) = e(çb) p+I = pk(dp)e(dp) n-2k 
=_ pk(r*TltI)- ,pe(¢)pk_e(d/) ±) e(çb) 
=0 
k-1 
 3k(Tr.Tl,l[){?(p)n-2k_(g..O3«(p)(3k_g(p.l_)3k(p) )= 
/ 
k--1 
-= p(r*TM)e(¢) - - ( Pe 
p2k_e (r*TltI) - _, Pe(dP)P2k-e-e(dP -I-) e(d/)) "-2-2 
g2:0 

= pk(r*TM)e(d/)) "-2 + T1 + T2 (modulo 2-torsion). 

TI -- (--1) s -pjs(7r*Tl,I)pI»(¢)e(dP) n-2k-2s, 
18 
(3.4) T2 = (-1) n°+l  pl,o+l(dp)p¢o(dp±)e(¢) "-2¢-2"°, 
Ino+l 
where no = [(n- 2k)/21 is the greatest integer not exceeding (n- 2k)/2, the sure 
in (3.3) runs over all multi-indices 
I»=(1,...,) with 0<_gl,...,g_<k-1 (l<_s_<[(n-2k)/2]=n0), 
the sure in (3.4) runs over all multi-indices 
I,o+1 = (,-.-,o+1) with 0 _< 1,---,no+1 -< k- 1, 
PI, = P«,pe " " " P«,, J» = (s+ l )k-( el +. . -+g), and k0 = (n0+l)k-(gl+-. 
Since et _< k-1 for all 1 < t < n0+l, we have k0 >_ n0+l > (n-2k)/2. Therefore, 
p¢o(¢ ±) and hence T2 vanishes. Furthermore, the degree of pl(dp)e(dp) '-2¢-2 is 
strictly smaller than 2k(n- 2k). Therefore, its image by the Gysin map rr! vanishes, 

Here we set 
(3.3) 
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and hence we have 7r!Tt = 0 by Lemma 2.1 (2). Thus, by Leinmas 3.3 and 2.1, we 
have 
rr!j!(e(j*¢) ) =_ pk(T.A1)7r(e(¢) n-ek) = pk(T.AI) 0nodulo 2-torsion). 
On the other hand, by (3.2) and Lemma 2.1 we have 
rr!j!(e(j*dp)) = i!!(e(j*dp)) = i!'!(e(*v)) (modulo 2-torsion) 
= i!(e(v)!(1)) = i!(e(v)). 
Thus, by definition of the self-intersection class, we get 
I(S(f)) = i!(e(oe)) = p(T.AI) (modulo 2-torsion). 
This completes the proof for the case N = R p. 
In the case of general N, we tan show the assertion similarly in a standard way, 
whi«h is sketched as follows (for more details, sec [12] for example). Take a vector 
bundle  over .AI such that f*TN (3  is trivial (for instance, taking an embedding 
of N into a Euclidean space of sufficiently high dimension, set.  to be the pull-back 
of the normal bundle of the embedding). Note that  defines the element -f*TN 
in the K-group. In t.he al»ove proof, T.A[ and dr (i.e., jf) should be replaced by 
T.AI (3  and df (3 i(l  T]tl (3  - f*TN (3 , respectively. Finally, we get 
i!(e(oe)) =_ pe(TM (3 ) =_ pn(T.AI - f*TN) (modulo 2-torsion). 
This completes the proof of Theorem 1.2. [] 

Remark 3.4. When the normal bundle v of the embedding i : S(f) "-, M is ori- 
entable, the Euler class e(u)  H2(S(f); Z) is a special case of the incident class 
defined by Rimnyi [24], i.e., e(u) = I(E 2, E2)(f). Note that our definition of the 
twisted Euler class uses the orientation local system so that everything works even 
if v is non-orientable. 

Now the formula (1.3) follows directly from Theorem 1.2, since (I(S(f)), [Ml) = 
S(f). S(f) by the definition of I(S(f)) and every orientable 3-manifold is paral- 
lelizable, where [.AI]  Ha(M: Z) is the fundamental class of M, and (,) is the 
Kronecker product. 

Rernark 3.5. When n -p + 1 is odd, e(u)  Hn-p+I(S(f); 0) is of order at most 
two, since the dimension of a fibre of u is odd. Hence I(S(f)) = i!(e(u)) is an 
element of order at most two or, in other words, it vanishes modulo 2-torsion. 

l?emark 3.6. Fehér and Rimnyi in [13] observed that if a singularity set r/(f) 
of type q is an orient.able closed submanifold of an orientable manifold, then the 
self-intersection class of r/(f) is equal to the Thom polynomial of the complexified 
singularity 'qc(fc) multiplied by (-1) '('-1)/2, where m is the (real) codimension 
of 'q. It seems possible to generalize this to the non-orientable case, as we have 
donc here for the singularity type AI. In fact, for complex analytic maps 
N'-2+ , the Thom polynomial for an A-type singularity coincides with c2 (TM- 
f*TN). We conjecture that our formula would be truc without taking modulo 2- 
torsion, and also for the case Et(f) # 0 for some  _> 2k + 1. 

Remark 3.7. For a generic map f  .AI -- N of a closed n-dimensional manifold 
into a p-dimensional manifold with  - p + 1 = 2k, k > 1, the Thom polynomial 
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Tp(Al(f)) coincides with the 2k-th Stiefel-Whitney class w2k(Tiil -- f*TN) E 
H2k(M; Z2) by Thom [311. Hence, we have 
I(S(f)) oe w2k(Tiil - f*TN)  (mod 2). 
On the other hand, we have the basic congruence 
w --p (mod 2) 
(see [21, Problem 15-AI, for example). This means t.hat the difference I(S(f))- 
pk(Tiil -- f*TN) is a multiple of two. 
Remark 3.8. In Theorem 1.2, we have assumed that the source manifold Iii is 
compact. However, this assumption is not necessary, as long as the map f : Iii , N 
s proper and we use the homology of closed support instead of the usual homology. 

4. NONEXISTENCE OF FOLD MAPS 
In this section, we give some necessary conditions for thc existence of fold maps 
as an application of our formula (1.2). Let us first recall the following. 
Definition 4.1. For a smooth mat) f  Iii ,/V with n = dira/I >_ dira N = p, a 
point q E Iii is a fold singularity (or an ,41 -type singularity) of f if f is of t.h(' form 
2 
(4.1) (Xl,... ,xn) H (x,... ,_,, ± ±... ± x) 
with respect to appropriate coordinates around q and f(q). Let A' be the number 
of negative signs appearing in the p-th component of the right-hand side of (4.1). 
Then, max{M, n-p+ 1 - M} is called the reduced index of q, which does hot depend 
on a particular choice of coordinates (for details, see [20]). If the singulal" set S(f) 
of f is empty or consists only of fold singularities, f is called a fold map.  
Now let us prove Theorem 1.3. 
Proof of Theorem 1.3. Suppose that there is a fold map f - Iii - N homotopic to g. 
For A = k, k+ 1 .... , n-p+l, let S(f) be the set of fold singularities of f of reduced 
index A. Note that S(f) is the disjoint union of Sk(f), Sk+(f) .... , S_p+(f) and 
that each S(f) consists of some connected components of S(f). As is easily ob- 
served, each S(f) is a (p- 1)-dimensional regular subinmfifold of Iii, aad 
is m imnmrsion for each . Furthermore, the normal bundle of the immersion is 
trivial for  ¢ k (for example, see [25]). This implies that Os(i) - (i)*f*ON for 
 ¢ k, where i  S(f) - Iii is the inclusion. Let. x  H2k(Iil; Of*TN--TM) : 
H2k(II; (-g*TN-TM) be the cohomology class Poincaré dual to the homology class 
in Hp-(M;f*ON) represented by S(f),   k. Then, the self-intersection ho- 
mology class S(f). S(f)  Hn-ak(M; OM) and the cohomology class x - x  
H ak (M; Z) are Poincaré dual to each other, and the self-intersection class I(S (f)) 
coincides with x - x. 
For  = k, let 'k be the normal bundle of Sk(f) in M. By [25], its structure 
group can be reduced to the semi-direct product (7 -=- (O(k) × O(k)) > Z2 C O(2k). 
Since O(k) × O(k) is a subgroup of G of index two, the structure group of the 
pull-back 7r,k of uk by an appropriate double covering 7ck  k(f) -- Sk(f) can be 
reduced to O(k) × O(k). This implies that 7cuk splits into the Whitney sure  
for some k-plane bundles  and 2 over k(f). 

lin [15], such a map is called a submersion with folds. 
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Since k is odd by our assumption, the twisted Euler classes of Ç1 and ç2 are 
elements of order at most two, and hence sois the Euler class e(rv) of rv. Since 
r!(e(rv)) = e(v)r!(1) = 2e(v), we sec that 4e(v) vanishes. This implies that 
t he self-intersection class I (Sk (f)) vanishes modulo 4-torsion. 

Since xA  xA, = 0 for A - A' and 
n--p+l 

A=k 
we conclude that the cohomology class 

satisfies 

n--p÷1 
Z z,X 6 H2/¢(/I; Og'TN-TM) 
A:k+1 

x  z =-- I(S(f)) (inodulo 4-torsion). 

Hence, by Theorem 1.2, the result follows. This completes the proof of Theorem 1.3. 

As an mportant corollary, we have the following, which shows the effectiveness 
of the formula (1.3). 
Corollary 4.2. Let Itl be a closed oriented 4-manifoid whose intersection fovm 
is isomorphic either fo +(I) or to +((i) (9 (i)). Then there exists n foid map 
f  M - N for any orientable 3-manifoid N. In other words, every generic map 
f  .AI - N necessarily has cusp singularities. 
Pro@ Suppose that there is a fold nlap f  BI -, N. Then by Theorem 1.3, there 
exists an element x 6 H2(/I; Z) such that 
x --. x = Pi (TM - f*TN) = Pi (M), 
since every orientable 3-manifold is parallelizable. When the intersection form of M 
is isomorphic to +(1), this implies that there exists an integer  such that 2 = 3, 
since Pi [.Al] = 3a(M). This is a contradiction. Similarly, when the intersection 
form of M is isomorphic to +((1) (9 (1)), there must exist integers 1 and '2 such 
that î +  = 6, which is a contradiction again. Hence, there exists no fold map 
f.M-N. [] 
Remark 4.3. In [27], the second author obtained the special case of Theorem 1.3 for 
(n, p) = (4, 3) by using a different method. Corollary 4.2 was also obtained there. 
(In fact, in [27], it was proved that the sufficient condition for the nonexistence of 
fold maps mentioned in Corollary 4.2 is also necessary.) Note that Corollary 4.2 
generalizes Theorem 1.1 and the main theorem of [30]. Note also that Akhmetiev 
and Sadykov [1] recently gave another proof of Theorem 1.1 from a slightly different 
point of view. 
By using Theorem 1.3, we also have the following result for general dimensions. 
Corollary 4.4. For every dimension pair (n,p) such that n -p + 1 = 2k for 
a positive odd integer k with 4k < n, there exsts a closed connected omentable 
manifold of dimension n that admits no fold map into any p-dimensionai manifold 
N such that pi(N) = 0 for all I < i < k. 
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Pro@ Set 

{ CP 2kxS n-4k, 
M = CP 2k-1 X S n-4k-l-2, 
Cp 2, 
Note that its cohonlology ring satisfies 
{ Z[a, b]/(a 2¢+1 , b2), 
H*(M; Z) - Z[a,b]/(a2k, b2), 
Z[a]/(a2+l), 

n-4k>2, 
n-4k=l, 
n = 4k. 

n - 4k _> 2, 
n-4k=l, 
n = 4k, 

where a corresponds to a generator of H2(CP2k; Z)  Z for n - 4k :/= 1 (or a 
generator of H2(CP 2k-l" Z)  Z for n - 4k = 1), and b corresponds to a generator 
of H'-4k(S'-4k; Z) = Z for n-4k _> 2 (or a generator of H'-4k+2(S'-4k+2; Z)  Z 
for n - 4k = 1). Note also that the total Pontrjagin class p(I) of I satisfies 
f (l+a2) 2k+, n-4k¢l, 
p(AI) 
(1 + a2) 2k, n - 4k = 1 
(for example, see [21]). 
Suppose that there exists a fold map f  M  N for some p-dilnensional manifold 
N such that pi(N) = 0 for 1 < i < k. Since M is simply connected, any local system 
over M is trivial. Then by Theorem 1.3, there exists an element x  H2k(M: Z) 
such that 
x  x = pk(TM - f*TN) = pk(M) =  
n 
4k 
1, 
(k)a2a, n - 4k : 1, 
since H*(M; Z) is torsion free. This implies that the integer 

(2k+ 1)(2k+ 1), (2) 
k - (k+l)!k! or --  
must be a square, which is a contradiction by a result of ErdSs [11] (see also [16]). 
Hence, M is a desired manifold. This completes the proof. [] 

Remark 4.5. In the above corollary, if n is even and k OE 3, then we can prove that 
Ci 9n/2 is also a desired manifold except possibly for (n, p) = (98, 93), since the 
binomial coeflïcient 
Çn/2k? 1) 
is a square if and only if n = 98 and k = 3 (see [16]). 

Corollary 4.6. Let (n,p) be a dimension pair such that n -p + 1 = 2k for a 
positive odd integer k with 4k < n. Then, for every closed oriented n-dimensional 
manifold M, there exists a closed connected oriented n-dimensional manifold M  
oriented cobordant to M such that 1I  admits no fold map into any p-dimensional 
manifold N with pi(N) = 0 for all 1 < i < k. 

Pro@ Let Mo be the n-dimensional manifold given by Corollary 4.4. Furthermore, 
let/I be a connected and simply connected manifold oriented cobordant to/I and 
set M  = hl 1llo1Io, where hlo denotes the manifold ]tfo with orientation reversed. 
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Then, since we have , > 4k by our assmnption, by all argument similar to that in 
the proof of the above corollary, we see that there exists no element a: E H2k(/1l': Z) 
such that x ,--. oc = pk(/1l') modulo 4-torsion. Hence, by Theorem 1.a,/11' admits 
uo fold map into N. Since/1I' is oriented cobordant to M, the result follows. [] 

Remark 4.7. Surprisingly enough, for (ri,p) = (4,3), the conclusion of the above 
corollary does not hold in general. In fact, if a closed oriented 4-manifold M has 
signature a(M) ¢ +1,+2, the,, /11 always dmits a fold map into 1R a (see [27]). 
Note that, in this case,/," = 1 and 4/,- = ,. 
Remark 4.8. I11 [19], it has been shown that if/11 is a closed manifold of odd Euler 
characteristic, then/11 canlmt adroit any fold map into 1R  for p ¢ 1, 3, 7 (see also 
[28]). V'e cm use this to obtain a result similar to Crollary 4.4 for other dimension 
pairs as well. However, such a result is hot useful for the proof of Corollary 4.6. 
Example 4.9. Let us consider the 4-dinmnsiolml complex projective space CP 4. By 
[19], if CP 4 admits a fold lnap into Dt I°, then p lnUSt be equal to 1,3 or ï, since 
CP 4 bas odd Euler characteristic. Clearly, it admits a fold map into 1R. However, 
it cannot adnlit a fold map into R r, since 
is not a square (for details, see the proof of Corollary 4.4 or Remark 4.5). We do 
not know if CP 4 admits a fold map into IR a or hot. 
Simitar observations hold also for CP 6, cP 2 x CP 2, HP 2, etc. (refer to [21] for 
the description of their Pontrjagin classes). Details are leff to the reader. 
Remark 4.10. Let f :/11 - N be a fold map of a closed orientable n-dimensional 
manifold into an orientable p-dimensiolml manifold such that ,,. - p + 1 = 2k. Let 
us suppose that the singular set S(f) of f is orientable. We give an arbitraw ori- 
entation to S(f) and let v E H2(/1I: Z) be the cohomology class Poincaré dual to 
the homology class represented by S(f). Then by [al], the modulo two reduction 
of v coincides with the Stiefel-Whitney class w2(T/1l - f*TN)  H2(M: Z). 
Furthermore, by Theorem 1.2, v ---. v = p(TM - f*TN) modulo 2-torsion. Sum- 
marizing, we see that there exists an eleiuent v  H2(/1I; Z) whose modulo two 
reduction coincides with w2k(T/1l - f*TN) Silch that v -- v =- p(T/1l - f*TN) 
modulo 2-torsion. 
The above result can also be proved as follows. By Ando [2], there exists a fold 
map f :/11 - N homotopic to a given snmoth map g :/11 - N such that S(f) is 
orientable if and only if there exists a fibrewise epimorphism ç : T/1I@e 1 -- g*TN, 
where K 1 is the t.rivial line bundle over _ai. Set r] = ker , which is an orientable 
vector bundle of rank n - p + 1 = 2k, and let v denote the Euler class of 
Then its modulo two reduction coincides with w2(TM - f*TN), and v -- v = 
p(TM - f*TN) (for details, see [21], for example). 
The above observation suggests that the formula (1.2) should be true without 
taking modulo 2-torsion. 

Remark 4.11. So far, we have obtained several nonexistence results for fold maps. 
For the existence results, refer to the works of Levine [20], Èliaberg [9], [10] and 
Alldo [2]. For the dimension pair (,z,p) = (4,3), see [27]. 
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ERRATUM TO 
"ARENS REGULARITY OF THE ALGEBRA AB" 

A. ULGER 

Professors Edmond Granirer and GrahaIn Colin have informed me that Theo- 
rem 4.20 in the last section of [1], which states that the projective tensor product 
C(K)  C(S) of two conmutative C*-algebras is Arens regular, is hot correct. The 
reason, as they have rightly told me, is that, if G is a compact group, then the alge- 
bra C(G)  C(G) contains an isonorphic copy of the Fourier algebra A(G), which 
is known to be not Arens regular. Consequently, C(G)  C(G) is hot Arens regular 
either. Unfortunately, this mistake also affects the subsequent results (from 4.21 to 
4.27) of the last section. By this erratum I acknowledge this mistake and let it be 
known to the people working iii this fiel& 
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ERRATUM TO 
"SPHERICAL CLASSES AND THE ALGEBRAIC TRANSFER" 

NGUYtN H. V. HU'NG 

Let QoS ° be the basepoint coinponent of QS ° = lim, çt"S '. .4 sphemcal class 
in QoS ° is an element belonging to the image of the Hurewicz homomorphisnl: 

7Es 0  
I . ,(s ) = ,(QoS °)  /,(çoS°). 

Here and throughout this note, homology is taken with coefficients in F2, the field 
of two elements. The long-standing conjecture on spherical classes reads as follows. 

Conjecture 1.1. There are no spherical classes in Q0S °, except the elements of 
Hopf invariant 1 and those of Kervaire invariant 1. 

An algebraic version of this problem goes as follows. Let t be a k-dimensional 
vector space over F2. Then, the polynomial algebra in k variables Pk = H* 
is a module over both the Steenrod algebra .A and the general linear group 
GL(k, F2). J. E. Larmes and S. Zarati constructed honlomorphisms 

k k+i PkGLk 

(see [6]) and have shown that these maps correspond to an associated graded of 
the Hurewicz homomorphism. The proof of this assertion is unpublished, but it is 
sketched by J. E. Lannes [5] and by P. G. Goerss [1]. The Hopf invariant 1 and the 
Kervaire invariant 1 classes are respectively represented by certain pernlanent cycles 
in Ext.1. (F2, F2) and Ext*(F2,F2), on which ç and ç2 are non-zero. Therefore, 
we are led to the following conjecture. 

Conjecture 1.2. ç = 0 in any positive stem i for k > 2. 

Iii the introduction of the article [2] we are mistaken in asserting that Lannes 
and Zarati's work shows that Conjecture 1.2 implies Conjecture 1.1. This cornes 
from the usual problem with spectral sequences: if an element maps to an element 
of higher filtration, then in the associated graded it will map to 0. Thus ç could 
be 0 even if H is not. Of course, if Conjecture 1.2 were false on a permanent cycle, 
then Conjecture 1.1 would also be false. 
Apart from this, all of our results and proofs in the article [2] are correct. 
This correction also applies to our papers [4] (joint with F. P. Peterson) and [3]. 
The author is grateful to Nick Kuhn for pointing out the above misunderstanding. 

Received by the editors February 4, 2003. 
2000 Mathematics Subject Classification. Primary 55P47, 55Q45, 55S10, 55T15. 

()2003 Americn Mathematical Society 

3841 



3842 NGUYIN H. V. HLrNG 

IEFERENCES 

1. P. G. Goerss, Unstable projectives and stable Ext: with apphcations, Proc. London Math. Soc. 
53 (1986), 539-561. MR 88d:55011 
2. N. H. V. Htmg, Sphemcal classes and the algebraic transfer, Trans. Amer. Math. Soc. 349 
(1997), 3893-3910. MR 98e:55020 
3. N. H. V. Htrng, The weak conjecture on spherical classes, Math. Zeit. 231 (1999), 727-743. 
MR 2000g:55019 
4. N. H. V. Htrng and F. P. Peterson, Spherical classes and the Dickson algebra, Math. Proc. 
Camb. Phil. Soc. 124 (1998), 253-264. MR 99i:55021 
5. J. Larmes, Sur le n-dual du n-ème spectre de Brown-Gitler, Math. Zeit. 199 (1988), 29-42. 
MR 89h:55020 
6. J. Larmes and S. Zarati, Sur les foncteurs dérivés de la alAstabilisation, Math. Zeit. 194 (1987), 
25-59. MR 88j:55014 

DEPARTMENT OF MATHEMATICS, VIETNAM NATIONAL UNIVERSITV, HANOI. 334 NGUVN TPI 
STREET, HANOI, VIETNAM 
E-mail address: nhvhungvnu, edu. vn 



Editorial Information 
To be published in the Transactions, a paper must be correct, new, nontrivial, and 
significant. Further, it must be well written and of interest to a substantial number of 
mathematicians. Piecemeal results, such as an inconclusive step toward an unproved 
major theorem or a minor variation on a known result, are in general not acceptable for 
publication. 
Papers submitted to the Transactions should exceed 10 published journal pages in 
length. Shorter papers may be submitted to the Proceedings of the Araerican Matheraatcal 
Society. Published pages are the same size as those generated in the style files provided 
for AjS-ITEX or AjS-TEX. 
As of May 31, 2003, the backlog for this journal was approximately 5 issues. This esti- 
mate is the result of dividing the number of manuscripts for this journal in the Providence 
office that have not yet gone to the printer on the above date by the average number 
of articles per issue over the previous twelve months, reduced by the number of issues 
published in four months (the time necessary for editing and composing a typical issue). 
In an effort to make articles available as quickly as possible, articles are posted to the 
AMS website individually after proof is returned from authors and before appearing in an 
issue. 
A Consent to Publish and Copyright Agreement is required before a paper will be 
published in this journal. After a paper is accepted for publication, the Providence office 
will send a Consent to Publish and Copyright Agreement to all authors of the paper. By 
submitting a paper to this journal, authors certify that the results have not been sub- 
mitted to nor are they under consideration for publication by another journal, conference 
proceedings, or similar publication. 
Information for Authors 
Initial submission. Two copies of the paper should be sent directly to the appropri- 
ate Editor and the author should keep a copy. IF an editor is agreeable, an electronic 
manuscript prepared in TEX or ITEX may be submitted by pointing to an appropriate 
URL on a preprint or e-print server. 
The first page must consist of a descriptive title, followed by an abstract that summa- 
rizes the article in language suitable for workers in the general field (algebra, analysis, etc.). 
The descriptive title should be short, but informative; useless or vague phrases such as 
"some remarks about" or "concerning" should be avoided. The abstract should be at least 
one complete sentence, and at most 300 words. Included with the footnotes to the paper 
should be the 2000 Mathematics Subject Classification representing the primary and sec- 
ondary subjects of the article. The classifications are accessible from u-. ams. org/r, sc/. 
The list of classifications is also available in print starting with the 1999 annual index 
of Matheraatical Reviews. The Mathematics Subject Classification footnote may be fol- 
lowed by a list of key words and phrases describing the subject matter of the article 
and taken from it. Journal abbreviations used in bibliographies are listed in the latest 
Mathematical Reviews annual index. The series abbreviations are also accessible from 
n.ams.org/publications/. To help in preparing and verifying references, the AMS 
offers MR Lookup, a Reference Tool for Linking, at r-. ams. org/mrlookup/. When the 
manuscript is submitted, authors should supply the editor with electronic addresses if 
available. These will be printed after the postal address at the end of each article. 
Electronically prepared manuscripts. The AMS encourages electronically pre- 
pared manuscripts, with a strong preference for ,4jS-ITF. To this end, the Society 
has prepared ,4jt4S-ITF author packages for each AMS publication. Author packages 
include instructions for preparing electronic manuscripts, the AMS Author Handbook, sam- 
ples, and a style file that generates the particular design specifications of that publication 
series. Articles properly prepared using the ,4jS-ITF style file and the \label and \ref 
commands automatically enable extensive intra-document linking to the bibliography and 
other elements of the article for searching electronically on the Web. Because linking 
must often be added manually to electronically prepared manuscripts in other forms of 
TF_, using ,4JS-ITF also reduces the amount of technical intervention once the files 



are received by the AMS. This results in fewer errors in processing and saves the author 
proofreading rime. AJ>IS-Dq-X papers also move more efficiently through the production 
stream, helping to minimize publishing costs. 
AJ>IS-DqX is the highly preferred format of TEX , but author packages are also available 
in AJ>IS-TF. Those authors who make use of these style files from the beginning of the 
writing process will further reduce their own efforts. Manuscripts prepared electronically 
in DTEX or plain TEX are normally not acceptable due to the high amount of technical rime 
required to insure that the file will run properly through the ANIS in-house production 
system. Dq_/X users will find that AJ>IS-DTF is the same as DTF_/X with additional 
commands to simplify the typesetting of mathematics, and users of plain TEX should have 
the foundation for learning AJ>IS-DTF. 
Authors may retrieve an author package from the AMS website starting from 
m4w. ams. org/tex/ or via FTP to ftp.ams, org (login as emonymous, enter username as 
password, and type cal pub/author-±nfo). The AMS Author Handbook and the In- 
struction Manual are available in PDF format following the author packages link from 
.ams.org/tex/. The author package can also be obtained free of charge by sending 
email to pubams, org (Interner) or from the Publication Division, American Mathematical 
Society, 201 Charles Street, Providence, 1I 02904-2294 USA. When requesting an author 
package, please specify AJ>IS-Dq-X or 42V/S-TEX, Macintosh or IBM (3.5) format, and 
the publication in which your paper will appear. Please be sure to include your complete 
mailing address. 
At the rime of submission, authors should indicate if the paper has been prepared using 
AJ>IS-DqX or AJ>IS-q-X and provide the Editor with a paper manuscript that matches 
the electronic manuscript. The final version of the electronic manuscript should be sent 
to the Providence office immediately after the paper has been accepted for publication. 
The author should also send the final version of the paper manuscript to the Editor, who 
will forward a copy to the Providence office. Editors will require authors to send their 
electronically prepared manuscripts to the Providence office in a timely fashion. Elec- 
tronically prepared manuscripts can be sent via email to pub-subm±tams.org (Interner) 
or on diskette to the Electronic Prepress Department, American Mathematical Society, 
201 Charles Street, Providence, RI 02904-2294 USA. When sending a manuscript elec- 
tronically, please be sure to include a message indicating in which publication the paper 
has been accepted. No corrections will be accepted electronically. Authors must mark 
their changes on their proof copies and return them to the Providence office. Complete 
instructions on how to send files are included in the author package. 
Electronic graphics. Comprehensive instructions on preparing graphics are available 
starting from .ams.org/jourhtml/authors.html. A few of the major requirements 
are given here. 
Submit files for graphics as EPS (Encapsulated PostScript) files. This includes graphics 
originated via a graphics application as well as scanned photographs or other computer- 
generated images. If this is not possible, TIFF files are acceptable as long as they can be 
opened in Adobe Photoshop or Illustrator. No marrer what method was used to produce 
the graphic, it is necessary to provide a paper copy to the AMS. 
Authors using graphics packages for the creation of electronic art should also avoid the 
use of any lines thinner than 0.5 points in width. Many graphics packages allow the user 
to specify a "hairline" for a very rhin line. Hairlines of*en look acceptable when proofed 
on a typical laser printer. However, when produced on a high-resolution laser imagesetter, 
hairlines become nearly invisible and will be lost entirely in the final printing process. 
Screens should be set to values between 15% and 85%. Screens which fall outside of this 
range are too light or too dark to print correctly. Variations of screens within a graphic 
should be no less than 10%. 
AMS policy on making changes to articles af*er posting. Articles are posted to 
the AMS website individually af*er proof is returned from authors and before appearing 
in an issue. To preserve the integrity of electronically published articles, once an article is 
individually posted to the AMS website but not yet in an issue, changes cannot be ruade 
in place in the paper. However, an "Add a,fter Dostin" section mav be added to the 



paper right before the References when there is a critical error in the content of the paper. 
The "Added after posting" section gives the author an opportunity to correct this type 
of critical error before the article is put into an issue for printing and before it is then 
reposted with the issue. The "Added after posting" section remains a permanent part of 
the paper. The AMS does hot keep author-related information, such as aflïliation, current 
address, and email address, up to date after a paper is initially posted. 
Once the article is assigned to an issue, even if the issue has hot yet been posted to the 
AMS website, corrections may be made to the paper by submitting a traditional errata 
article to the Editor. The errata article will appear in a future print issue and will link 
back and forth on the web to the original article online. 
Secure manuscript tracking on the Web and via email. Authors can track their 
manuscripts through the AMS journal production process using the personal AIIS ID and 
Article ID printed in the upper right-hand corner of the Consent to Publish form sent to 
each author who publishes in AhIS journals. Access to the tracking system is available 
from m¢w.ams, org/mstrack/ or via email sent to mstrack-queryCams.org. To access by 
email, on the subject line of the message simply enter the AIIS ID and Article ID. To 
track more than one manuscript by email, choose one of the Article IDs and enter the 
AMS ID and the Article ID followed by the word ali on the subject line. An explanation 
of each production step is provided on the web through links from the manuscript tracking 
screen. Questions can be sent. to tran-queryCams, org. 
TEX files available. Beginning with the January 1992 issue of the Bulletin and the 
January 1996 issues of Transactions, Proceedings, Mathematics of Computation, and the 
Journal of the AMS, TEX files can be downloaded from the AhIS website, starting from 
.ams.org/jouxnals/. Authors without Web access may request their files at the 
address given below after the article has been published. For Bulletin papers published in 
1987 through 1991 and for Transactions, Proceedings, Mathematics of Computation, and 
the Journal of the AMS papers published in 1987 through 1995, TEX files are available 
upon request for authors without Web access by sending email to f±le-requestCams, org 
or by contacting the Electronic Prepress Department, American hlathematical Society, 
201 Charles Street, Providence, RI 02904-2294 USA. The request should include the title 
of the paper, the naine(s) of the author(s), the naine of the publication in which the paper 
has or will appear, and the volume and issue numbers if known. The TEX file will be sent 
to the author making the request after the article goes to the printer. If the requestor can 
receive Internet email, please include the email address to which the file should be sent. 
Otherwise please indicate a diskette format and postal address to which a disk should 
be mailed. Note: Because TEX production at the AIIS sometimes requires extra fonts 
and macros that are hot yet publicly available, TEX files cannot be guaranteed to run 
through the author's version of TEX without errors. The ANIS regrets that it cannot 
provide support to eliminate such errors in the author's TEX environment. 
Inquiries. Any inquiries concerning a paper that has been accepted for publication 
that cannot be answered via the manuscript tracking system mentioned above should be 
sent to tran-queryams, org or directly to the Electronic Prepress Department, American 
Mathematical Society, 201 Charles Street, Providence, RI 02904-2294 USA. 



Editors 
The traditional method of submitting a paper is to send two hard copies to the appro- 
priate editor. Subjects, and the editors associated with them, are listed below. 
In principle the Transactions welcomes electronic submissions, and some of the editors, 
those whose names appear below with an asterisk (*), have indicated that they prefer 
them. Editors reserve the right to request hard copies after papers bave been submitted 
electronically. Authors are advised to make preliminary inquiries to editors as to whether 
they are likely to be able to handle submissions in a particular electronic form. 
Algebraic geometry, DAN ABRAMOVICH, Departrnent of Mathernatics, Boston Univer- 
sity, 111 Curnrnington Street, Boston, MA 02215 USA; e-mail: abraovi«¢bu.edu 
Algebraic topology and cohomology of groups, STEWART PRIDDY, Departrnent of 
Mathernatics, Northwestern University, 2033 Sheridan Road, Evanston, IL 60208-2730 USA; 
e-rnail: pr±ddyCraath, nwu. edu 
* Combinatorics. SERGEY FOMIN, Departrnent of Mathernatics, East Hall, University of 
Michigan, Ann Arbor, MI 48109-1109 USA; e-mail: fora±nCum£ch.edu 
Comple× analysis and geometry, D. H. PHONG, Departrnent of Mathernatics, Colurnbia 
University, 2990 Broadway, New York, NY 10027-0029 USA; e-rnail: phongmath, columb£a.edu 
* Differential geometry and global analysis, LISA C. JEFFREY, Departrnent of Math- 
ernatics, University of Toronto, 100 St. George Street, Toronto, Ontario, Canada MSS 3G3; 
e-mail: j effreymath, toronto, edu 
Dynamical systems and ergodic theory, ROBERT F. WILLIAMS, Departrnent of Math- 
ernatics, University of Texas, Austin, TX 78712-1082 USA; e-mail: bobmath.utexas.edu 
* Geometric analysis, TOBIAS COLDING, Courant Institute, New York University, 251 
Mercer Street, New York, NY 10012 USA; e-mail: cold±ng¢cms.nyu.edu 
Harmonic analysis, ALEXANDER NAGEL, Departrnent of iMathernatics, University of W'is- 
consin, 480 Lincoln Drive, Madison, Wl 53706-1313 USA; e-mail: nagelmath.w£sc.edu 
Harmonic analysis, representation theory, and Lie theory, ROBERT J. STANTC)N, 
Departrnent of Mathernatics, C)hio State University, 231 West 18th Avenue, Colurnbus, OH 43210- 
1174 USA; e-mail: stantonmath, ohio-state, edu 
iNumber theory, HARC)LD G. DIAIVIOND, Departrnent of Mathernatics, University of llli- 
nois, 1409 West Green Street, Urbana, IL 61801-2917 USA; e-mail: diamondmath, uuc. edu 
* Ordinary differential equations, partial differential equations, and applied math- 
ematics, PETER W. BATES, Departrnent of Mathernatics, Michigan State University, East 
Lansing, MI 48824-1027 USA; e-mail: batesCraath.rasu.edu 
* Partial differential equations, PATRICIA E. BAUMAN, Departrnent of Mathernatics, 
Purdue University, West Lafayette, IN 47907-1395 USA; e-mail: baumanmath.purdue.edu 
* Probability and statistics, KRZYSZTOF BURDZY, Departrnent of Mathernatics, Univer- 
sity of Washington, Box 354350, Seattle, WA 98195-4350 USA; e-mail: burdzymath.:ash£ngton. 
edu 
* Real analysis and partial differential equations, DANIEL TATARU, Departrnent of 
Mathernatics, University of California, Berkeley, CA 94720 USA; e-mail: tatarumath.berkeley. 
edu 
Ail other communications to the editors should be addressed to the Managing Editor, 
WILLIAM BECKNER, Departrnent of Mathernatics, University of Texas, Austin, TX 78712-1082 
USA; e-mail: becknermath, utexas, edu 

MEMOIRS OF THE AMERICA1N MATHEMATICAL SOCIETY 

Memoirs is devoted to research in pure and applied mathematics of the same nature as 
Transactions. An issue consists of one or more separately bound research tracts for which 
the authors provide reproduction copy. Papers intended for Memoirs should normally be 
at least 80 pages in length. Memoirs has the saine editorial committee as Transactions; 
so such papers should be addressed to one of the editors listed above. 



( Colinued frornback co,er) 
Toru Ohmoto, Osamu Saeki, and Kazuhiro Sakuma, Self-intersection 
class fi»r singularities and its application to fold maps .............. 3825 
A. llger, Erratum to "Arens regularity of the algebra A .- B'" . ........ 383.0 
Nguyn H. V. Htrng, Erratum to "'Sl)llerical classes and the algcl)raic 
transfer'" . ........................................................... 3,41 



TRANSACTIONS OF THE AMERICAN MATHEMATICAL SOCIETY 
CONTENTS 
Vol. 355, No. 9 Whole No. 820 September 2003 

Heinz H. Bauschke, Frank Deutsch, Hein Hundal, and Sung- 
Ho Park, A('('(dcrating the ('onvergen('e of the method of alternating 
l)roj(('ti(»ns .......................................................... 3433 
Sunghan Bae, Ernst-Ulrich Gekeler, Pyung-Lyun Kang, and Lin- 
sheng Yin, Aml(,rsou's (louifl« ('Oml)l«x and gamma monomials for 
ratiomd flm('tiou fi«'hls ............................................. 3463 
Lucia Caporaso, R(,mm'ks ai)out unifi)rm i)(nm(lc(hmss of ratiolml points 
over fm('lion fi'hls ......................................... 3475 
Thomas Keilen» [rr(,(lu«ii»ilily of «(luisingular familics of ('urvcs .......... 3485 
L. Brandolini, A. Iosevich» and G. avaglini, Planar ('onvex i)odics. 
i»m'i('r trausfiwm, latti('(' i»«)iuts, and irr('gularities of (listlii)ution .... 3513 
Shangbin Cul and Avner iedman, A ri'ce i»(»un(lary i)l'(»i)lcm for a 
sinulm" syst«'m of (lifl,r(,ltial (,(luations: An ai)l)li('ation to a model of 
t m«)r growth ................................................. 3537 
José Garcia-Cuerva, José iamlel Marco, and Javier Pcet, Shar I) 
I-i»mi(,r tyi»(, and ('otyl»(' with r('Sl)('('t te» ('omi)a('t scmisilnple Lie 
gr(»Ul»S ................................................... 3591 
J. Rosick and W. Tholen, Let)-(let(rmin('(l nodel categories and Ulfiversal 
h(»mOtOlS"  h('(»ri('s ............................................... 3611 
Christian Henriksen, Tire ('()mi)inat(»l'ial rigidity çOlljectlil'C is false for cui)ic 
l)olynomials ................................................. 3625 
Leo T. Butler, Z(,l'O (,utrol)y. u«m-int('glable g('odesic flows ald a non- 
('(»mmutativ(' rotation vc('tor . ....................................... 3641 
Carlos Sancho de Salas C(»mpl('te homogen(,(ms varieti(,s: Structure and 
('lassifi('ation ......................................................... 3651 
Neil O'Connell, A l)ath-trasfi»rmation for random walks and the Roi)insol- 
S('h(,nst('d ('orr(Sl)on(l('n('e ........................................... 3669 
Tae Tsuji, ()n th(, Iwasawa A-invariants (»f rcal ai)elian tic'lois ........... 3699 
Xiaoxiang Jiao alld Jiagui Peng, Ps('u(l(»-h(»h»ll(»rI)hi(' ('urves iii ('omplex 
GlaSsmami llmdfifl(ls ............................................... 3715 
Vassilis G. Papanicolaou, The i)(,rio(li(  Eul(,r-Beruoulli e(luation ....... 3727 
ancisco Jesds Castro-Jiménez and Nobuki Tayama, Singularities 
of the hyI)erg('ometri(" syst('m asso('iate(l with a lllOnonlial cmve .... 3761 
Jason P. Bell and Stanley N. Burris, Asyml)totics for logical limit laws: 
Vhel the growth of the ('omponents is in an RT class ............... 3777 
Michael E. Hoffman, Combilmtolics «)f rooted trees and Hol)f alge- 
bras ............................................................... 3795 
Mahuya Datta, Commctions with I)res('rii)ed first Pontagin fOrlll ....... 3813 
( Continued on inside back cover) 

888Z-9947(Z88389)355: 



American Mathematical Society.Transactions 
v.355, no.7-9 ; 2003 



